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ABSTRACT 

The  aim  of  this  research  was  to  investigate  the  effect  of  different  user  interface  designs  on  the 
performance  of  an  Unmarmed  Underwater  Vehicle  (UUV)  pilot.  Participants  in  this  study  were  23 
males  and  3  females  who  took  part  in  a  remote  piloting  experiment.  Participants  were  each 
presented  with  three  display  designs;  a  display  analogous  to  the  current  Mine  Disposal  Vehicle 
(MDV)  Baseline  display,  an  Inside-Out  (fixed  vehicle)  design  and  an  Outside-ln  (moving  vehicle) 
design  and  were  asked  to  fly  a  simulated  mission.  During  each  condition.  Situation  Awareness 
(SA)  and  Human  Performance  (HP)  measurements  were  taken.  Results  indicated  a  significant 
relationship  between  display  design  and  level  of  situation  awareness  and  human  performance  on  a 
number  of  measures.  Significant  differences  in  situation  awareness  were  observed  between  display 
designs  for  vehicle  roll  and  depth.  Results  also  indicated  significant  differences  between  the 
display  designs  for  the  number  of  control  reversal  errors  observed  for  roll,  the  number  of 
waypoints  reached,  the  final  odometer  reading  and  the  speed  of  approach  to  the  first  waypoint.  A 
significant  preference  was  revealed  for  the  Outside-ln  display  design.  Results  from  this  study 
indicate  that  UUV  pilot  situation  awareness  and  performance  can  be  enhanced  by  modifying  and 
improving  display  design.  Results  of  this  study  have  implications  for  the  use  of  unmanned 
vehicles  in  the  wider  air  and  land  domains,  as  well  as  the  underwater  domain. 
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Investigating  the  Effects  of  Display  Design  on 
Unmanned  Underwater  Vehicle  Pilot  Performance 


Executive  Summary 

Over  recent  years.  Unmanned  Vehicles  (UVs)  have  emerged  as  a  sensible  and  practical 
alternative  to  human  involvement  in  many  situations.  As  such,  these  vehicles  have 
been  embraced  by  commercial  and  military  establishments  around  the  world. 
Increasingly,  UVs  of  all  forms  are  taking  on  roles,  particularly  in  the  military 
environment,  where  the  risk  to  human  life  is  considered  too  high  or  where 
environmental  conditions  are  too  inhospitable  for  humans. 

As  part  of  the  UV  family.  Unmanned  Underwater  Vehicles  (UUVs)  constitute  an  area 
of  growing  interest  due  to  their  ability  to  operate  at  depths  and  in  areas  that  are 
inaccessible  to  humans  or  other  types  of  vessels.  These  vehicles  are  used  extensively  for 
underwater  search  and  salvage,  inspection,  surveying,  scientific  exploration,  and  mine 
countermeasures.  Owing  to  their  operational  capabilities,  the  Royal  Australian  Navy 
(RAN)  employs  UUVs  to  complete  a  wide  range  of  tasks,  including  mine  counter¬ 
measure  applications  and  hydrographic  surveillance  operations.  Due  to  the  important 
roles  these  vehicles  assume  in  such  situations,  it  is  necessary  to  ensure  superior 
systems  and  technologies  are  in  place  in  order  to  facilitate  optimal  performance  -  by 
both  human  and  machine. 

Unique  in  terms  of  their  operation,  UUVs  are  operated  from  a  location  external  to  the 
pilot's  present  position.  The  ultimate  goal  of  any  unmarmed  system  is  to  provide  a 
pilot  with  the  capability,  via  the  use  of  sensors,  to  act  and  perform  as  if  he  or  she  was 
really  present  at  the  remote  location.  In  support  of  this,  it  is  necessary  to  have  correct 
and  relevant  information  to  aid  completion  of  an  operational  task. 

The  Human  Computer  Interface  (HCl)  used  in  the  operation  of  a  UUV  is  of  particular 
importance  as  it  has  the  potential  to  influence  pilot  performance  and  degrade  mission 
effectiveness.  Due  to  the  complexity  of  operating  this  type  of  vehicle,  special 
consideration  needs  to  be  made  when  assessing  pilot  information  requirements,  and 
the  subsequent  design  and  presentation  of  information  displays.  Information  such  as; 
pitch,  roll,  heading,  depth,  speed  and  vehicle  location  can  be  considered  critical  to  the 
task  at  hand  and  as  such,  pilot  operator  displays  must  present  this  information  in  a 
marmer  which  is  accurate,  accessible  and  easily  interpretable. 

The  integration  and  presentation  of  vital  information  helps  provide  an  overall 
understanding  to  the  pilot  of  their  position  within  the  context  of  their  operational 
environment.  This  understanding  can  be  described  in  terms  of  a  pilot's  level  of 
Situation  Awareness  (SA).  As  a  concept,  SA  describes  "the  perception  of  elements  in 
the  environment  within  a  volume  of  time  and  space,  comprehension  of  their  meaning 
and  projection  of  their  status  into  the  future". 

Pilot  SA  is  an  essential  requirement  for  the  operation  of  a  UUV  as  a  lack  of  SA  has  the 
potential  to  greatly  influence  the  success  of  operations.  Display  design  becomes 


particularly  important  when  considering  SA  within  a  complex  system  such  as  a  UUV 
due  to  its  potential  influence  on  performance.  If  a  display  does  not  provide  a  pilot  with 
adequate  information  in  a  manner  which  is  conducive  to  achieving  and  maintaining  a 
high  level  of  SA,  then  performance  within  the  system  may  suffer.  It  is  therefore 
necessary  to  experimentally  examine  UUV  display  design  to  ensure  it  facilitates 
superior  performance. 

The  current  study  was  based  on  research  conducted  in  the  aviation  domain  assessing 
pilot  information  displays.  The  aim  was  to  extend  on  this  research  assessing  its 
relevance  and  application  to  the  operation  of  a  UUV. 

Conducted  as  a  milestone  under  task  02/128  Applications  of  ROV  technologies  and 
also,  in  conjunction  with  the  Department  of  Psychology  at  Monash  University,  the 
present  study  sought  to  investigate  the  effects  of  different  user  interface  designs  on 
Unmanned  Underwater  Vehicle  pilot  performance.  A  simulation  program  was 
specifically  developed  for  this  study  which  enabled  participants  to  take  part  in  a 
remote  piloting  experiment.  Participants  were  asked  to  fly  three  simulated  missions, 
each  time  presented  with  one  of  three  display  designs;  a  design  analogous  to  the 
current  Mine  Disposal  Vehicle  (MDV)  Baseline  display,  a  display  developed  based  on 
the  Inside-Out  (fixed  vehicle)  principle  of  display  design,  and  a  display  developed 
based  on  the  Outside-ln  (moving  vehicle)  principle  of  display  design.  During  each 
condition,  SA  and  human  performance  measurements  were  taken. 

Data  was  analysed  using  a  standard  Analysis  of  Variance  approach  coupled  with  post- 
hoc  tests  in  order  to  determine  the  presence  and  strength  of  any  relationship  between 
display  design  and  pilot  performance.  Results  indicated  a  significant  relationship 
between  display  design  and  level  of  situation  awareness  and  human  performance  on  a 
number  of  measures  indicating  that  UUV  pilot  situation  awareness  and  performance 
can  be  enhanced  by  modifying  and  improving  pilot  display  design.  Results  of  this 
study  have  implications  for  the  use  of  unmanned  vehicles  in  the  wider  air  and  land 
domains,  as  well  as  continued  application  in  the  underwater  domain.  The  results  of 
this  study  will  be  discussed  in  full  in  this  report. 
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1.  Introduction 


Designing  an  operator  display  for  the  operation  of  a  UUV  requires  close  consideration  of 
many  human  factors  issues.  Display  design  is  of  particular  importance  in  UUV  operations 
as  it  has  the  potential  to  influence  pilot  performance  and  degrade  mission  effectiveness. 
UUVs  are  distinctive  in  terms  of  their  operation  as  essentially,  they  are  operated  from  a 
location  external  to  a  pilot's  present  position.  Due  to  the  complexity  of  operating  this  type 
of  vehicle,  special  consideration  needs  to  be  made  when  assessing  pilot  information 
requirements,  and  the  subsequent  design  and  presentation  of  information  displays. 

From  a  human  factors  perspective,  the  study  of  display  design  is  not  novel.  Much  research 
has  been  conducted  in  the  aviation  domain  examining  pilot  display  designs  [1-8]  with  the 
intention  of  improving  system  -  both  man  and  machine  -  performance.  However,  these 
studies  have  been  conducted  for  the  purpose  of  application  to  piloted  aircraft,  not 
unmanned  aircraft.  While  a  number  of  studies  have  been  conducted  for  Unmanned  Air 
Vehicles  (UAVs),  they  have  tended  to  focus  on  system  technologies  [9-12]  largely 
neglecting  to  address  critical  human  factor  issues.  A  small  amount  of  UAV  human  factors 
research  has  been  conducted  by  the  United  States  Air  Force  examining  the  operator 
workstation  for  the  Global  Hawk  UAV  however,  this  research  has  not  specifically 
addressed  issues  associated  with  operator  display  design  [13]. 

In  contrast  to  UAVs,  UUVs  have  enjoyed  little  to  no  human  factors  research,  particularly 
with  reference  to  display  design.  Currently,  UUV  pilot  displays  have  been  developed  and 
implemented  principally  favouring  a  numerical  format  which  is  by  no  means  ideal  [14].  As 
minimal  research  into  display  design  exists  in  the  UAV  domain,  and  as  UUVs  operate  in  a 
vastly  different  environment,  it  is  necessary  to  consider  display  design  issues  as  they 
pertain  to  the  operation  of  this  class  of  vehicle  in  their  own  right  [13]. 

Like  their  air  counterparts,  UUV  pilots  require  particular  information  to  perform  their  job 
effectively.  The  ultimate  goal  of  any  unmarmed  system  is  to  provide  a  pilot  with  the 
capability,  via  the  use  of  sensors,  to  act  and  perform  as  if  he  or  she  was  really  present  at 
the  remote  location  [15].  In  support  of  this,  it  is  necessary  to  have  correct  and  relevant 
information  to  aid  completion  of  an  operational  task  [16].  Information  such  as;  pitch,  roll; 
heading;  depth;  speed  and  vehicle  location  can  be  considered  critical  to  the  task  at  hand 
and  as  such,  pilot  operator  displays  must  present  this  information  in  a  manner  which  is 
precise,  accessible  and  easily  interpretable.  The  integration  and  presentation  of  vital 
information  helps  provide  an  overall  understanding  to  the  pilot  of  their  position  within 
the  context  of  their  operational  environment.  This  understanding  can  be  described  in 
terms  of  a  pilot's  level  of  Situation  Awareness  (SA).  As  a  concept,  situation  awareness 
describes  "the  perception  of  elements  in  the  environment  within  a  volume  of  time  and 
space,  comprehension  of  their  meaning  and  projection  of  their  status  into  the  future"  [17, 
18]. 


Pilot  situation  awareness  is  an  essential  requirement  for  the  operation  of  a  UUV  as  a  lack 
of  SA  has  the  potential  to  greatly  influence  the  success  of  operations  [19].  Display  design 
becomes  particularly  important  when  considering  SA  within  a  complex  system  such  as  a 
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UUV  due  to  its  potential  influence  on  performance.  If  a  display  does  not  provide  a  pilot 
with  adequate  information  in  a  manner  which  is  conducive  to  achieving  and  maintaining  a 
high  level  of  SA,  then  performance  within  the  system  may  suffer. 

The  effectiveness  of  a  display  can  be  measured  by  a  pilot's  level  of  SA  and  subsequent 
human  performance.  While  SA  is  included  under  the  broader  heading  of  human 
performance,  it  can  also  be  considered  a  separate  measure  which  can  be  correlated  against 
other  performance  measures.  In  many  socio-technical  systems,  different  aspects  of  human 
performance,  such  as  time  on  task  and  accuracy,  can  be  measured  directly  from  system 
data  output.  An  operator's  level  of  SA  can  be  determined  through  a  measure  of  SA  and 
correlated  against  actual  system  data  to  get  a  feel  for  how  SA  affects  different  aspects  of 
performance.  Human-in-the-loop  simulation  provides  the  opportunity  to  evaluate  display 
design  with  a  view  to  determining  display  components  that  enhance  SA  and  improve  pilot 
performance  [19].  The  present  study  sought  to  investigate  ways  to  display  information  to 
pilots  that  would  enhance  SA  and  improve  pilot  performance. 

1.1  Inside-Out  and  Outside-In  Display  Design  Principles 

Piloting  a  UUV  can  be  a  very  difficult  task.  Pilots  have  to  be  sufficiently  skilled  to  operate 
the  vehicle  to  a  performance  level  that  is  commensurate  with  actually  being  on  the  vehicle. 
So  when  considering  the  complexity  of  UUV  operations,  the  pilot's  frame  of  reference 
becomes  important  in  determining  how  to  display  critical  information. 

Consider  the  example  of  display  symbol  motion.  When  determining  the  most  appropriate 
way  to  display  information  to  a  pilot,  it  is  necessary  to  address  the  question  of  whether 
they  perceive  the  display  as  representing  their  vehicle  moving  within  the  external  world, 
or  whether  they  perceive  the  external  world  as  moving  around  their  vehicle.  These  two 
basic  movement  relationships  can  be  described  as  an  Inside-Out  (egocentric  or  vehicle 
referenced)  frame  of  reference;  or  an  Outside-In  (exocentric  or  world  referenced)  frame  of 
reference;  the  difference  being  whether  the  vehicles  co-ordinates  are  used  as  the  reference 
system,  or  the  earth's  coordinates  are  used  [20].  Both  these  frames  of  reference  are  used 
within  the  aviation  domain  to  display  information  to  pilots  [21]. 

An  Inside-Out  display  design  describes  the  frame  of  reference  typically  used  in  most 
aircraft  displays  [21].  Most  commonly,  the  term  Inside-Out  refers  to  the  depiction  of  pitch 
and  roll  information.  An  artificial  moving  horizon  display  is  used  which  lines  up  with  the 
horizon  that  a  pilot  views  while  looking  straight  ahead.  When  a  pilot  banks  a  plane  right, 
the  artificial  horizon  rolls  to  the  left  mimicking  the  motion  of  the  true  horizon  (Figure  1). 
With  this  design,  a  small  aircraft  or  vehicle  symbol  remains  stationary  in  the  middle  of  the 
display  and  is  used  as  a  reference  point.  The  same  relationship  also  applies  to  information 
relating  to  climbing  and  descending,  with  an  aircraft's  pitch  information  typically 
displayed  by  a  pitch  ladder  which  indicates  the  aircrafts  position  above  or  below  a  given 
point.  This  moving  horizon  symbology  is  approved  by  the  Federal  Aviation 
Administration  and  is  a  recognised  international  standard  by  most  nations  [7,  21]. 
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Figure  1:  Inside-Out  depiction  of  roll  [20] 

In  contrast,  an  Outside-In  design  describes  another  frame  of  reference  which  can  be 
adopted  to  display  aircraft  information  to  a  pilot.  This  display  is  used  most  by  countries 
who  fly  with  Russian-built  aircraft  [21].  The  Outside-In  or  moving  aircraft  display 
typically  depicts  roll  information  directly  opposite  to  that  of  the  Inside-Out  approach.  In 
the  Outside-In  display,  the  horizon  remains  stationary  in  the  display  case  while  a 
miniature  aircraft  symbol  rolls  within  the  display  (Figure  2).  When  a  pilot  banks  a  plane  to 
the  right,  the  miniature  aircraft  rolls  to  the  right  accordingly.  The  Outside-In  display  also 
displays  pitch  information  using  a  type  of  pitch  ladder.  Various  researchers  have  reported 
that  the  Outside-In  display  tends  to  be  more  intuitive  than  the  moving  horizon  display  as 
it  mimics  stick  input  and  therefore  may  cause  fewer  control-reversal  errors  [22,  23]. 


Figure  2:  Outside-In  depiction  of  roll  [20] 

Essentially  other  information  relevant  to  a  UUV  pilot,  such  as  depth  and  heading,  can  also 
be  displayed  to  the  pilot  in  either  an  Inside-Out  or  Outside-In  manner. 
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Information  on  vehicle  depth  can  be  displayed  in  an  Inside-Out  manner  using  a  fixed 
pointer-moving  scale  representation.  In  contrast,  this  information  can  also  be  displayed  in 
an  Outside-In  way  with  a  fixed  scale-moving  pointer  display.  Heading  information  can 
also  be  displayed  in  a  similar  fashion  with  information  being  vehicle  referenced  or  world 
referenced  (Figure  3).  The  costs  and  benefits  associated  with  each  design  become  visible 
when  assessing  the  level  of  performance  and  SA  that  pilots  can  achieve. 


Figure  3:  Two  principles  for  displaying  vehicle  information  and  the  resulting  display  depiction: 

(a)  Inside-Out  or  fixed  pointer  depth  display;  (b)  Outside-In  or  moving  pointer  depth 
display;  (c)  Inside-Out  or  fixed  pointer  heading  display;  (d)  Outside-In  or  moving 
pointer  heading  display 


The  Outside-In  design  concept  has  been  the  subject  of  much  interest  and  research 
comparing  its  capabilities  relative  to  Inside-Out  displays.  Most  of  this  research  was 
conducted  prior  to  the  1970s  as  described  in  an  extensive  review  article  by  Johnson  and 
Roscoe  (1972)  and  supports  the  use  of  these  displays  in  the  aviation  environment. 
Nevertheless,  Inside-Out  displays  have  continued  to  be  used  extensively  in  the  aircraft 
used  in  the  Western  world  up  until  the  present  day. 


The  current  study  is  based  on  research  conducted  in  the  aviation  domain  assessing  pilot 
information  displays  [22].  The  aim  was  to  extend  on  this  research  and  assess  its  relevance 
and  application  to  the  operation  of  a  UUV.  The  present  study  sought  to  investigate  the 
effects  of  display  design  on  UUV  pilot  performance.  Additional  literature  in  the  aviation 
domain  has  demonstrated  the  relative  influence  of  display  design  on  situation  awareness 
and  human  performance  [24-26].  In  line  with  this,  the  current  study  aimed  to  examine 
different  frames  of  reference  for  displaying  information  to  pilots  of  unmanned  underwater 
vehicles  hypothesising  Display  Design  will  influence  a  pilot's  level  of  SA  with  the  Outside- 
In  design  facilitating  higher  levels  of  SA.  It  was  also  hypothesised  that  Display  Design 
would  affect  the  performance  measures  of  the  number  of  control  reversal  errors,  number 
of  waypoints  reached  and  the  total  distance  travelled  with  the  Outside-In  design  again 
facilitating  superior  performance  over  the  remaining  displays.  The  results  of  this  study 
will  be  discussed. 
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2.  Experiment 


2.1  Method 

2.1.1  Participants 

Participants  in  the  study  were  26  individuals  recruited  from  the  Royal  Australian  Navy's 
Mine  Warfare  Faculty  and  the  Defence  Science  and  Technology  Organisation.  The  sample 
included  experienced  Unmanned  Underwater  Vehicle  and  fixed  wing  aerial  pilots,  as  well 
as  flight  naive  participants.  The  data  obtained  were  representative  of  23  Males  and  3 
Females  who  ranged  in  age  from  23  to  63  years,  contributing  a  mean  age  of  39.76  years  and 
a  standard  deviation  of  11.29  years.  Participants  took  part  in  all  three  of  the  experimental 
conditions  and  participation  was  on  a  volunteer  basis. 

2.1.2  Materials 

The  experiment  was  conducted  via  the  use  of  a  computer  simulation  program  specifically 
developed  for  this  study.  This  program  was  developed  extensively  over  a  two  year  period. 
For  information  on  the  development  of  this  program,  please  contact  the  first  author. 

The  simulation  program  ran  on  a  Pentium  4  -  1.80GHz  Dell  Laptop  computer.  The  laptop 
computer  was  under  the  control  of  the  experimenter  who  was  present  for  the  rurming  of 
the  experiment.  The  laptop  was  connected  to  an  external  SAMPA  colour  monitor  (48.26cm 
diagonal  measurement)  that  sat  in  front  of  the  participants,  and  which  they  used  to  view 
the  displays  in  each  of  the  experimental  conditions. 

Participants  controlled  a  simulated  UUV  with  a  joystick  that  allowed  six  degrees  of 
freedom.  The  joystick,  which  was  attached  to  the  laptop  computer,  was  situated  in  front  of 
the  participants  colour  monitor  and  was  used  to  control  the  movements  of  the  vehicle 
(information  on  joystick  control  is  provided  in  Appendix  A).  Figure  4  shows  the 
experimental  setup. 


Figure  4:  Experimental  set  up 
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Human  performance  and  situation  awareness  data  were  collected.  Human  performance 
data  were  collected  and  logged  directly  by  the  simulation  program.  Situation  awareness 
data  was  also  collected  electronically  through  the  simulator  which  incorporated  the 
Situation  Awareness  Global  Assessment  Technique  (SAGAT)  [18, 19]  which  is  designed  to 
question  an  operator's  level  of  situation  awareness.  The  SAGAT  queries  were  developed 
following  the  completion  of  a  Goal  Directed  Task  Analysis  (GDTA)  [27].  A  comprehensive 
explanation  of  the  SAGAT  and  GDTA  will  be  provided  in  the  procedure  section  of  this 
report  with  further  information  located  in  the  relevant  appendices. 

2.1.3  Procedure 

Prior  to  the  design  and  commencement  of  this  experiment,  significant  preliminary  work 
was  conducted  which  included  conducting  a  number  of  interviews  and  focus  group 
sessions  with  experienced  UUV  pilots.  The  interviews  and  focus  group  sessions  were 
conducted  over  one  day  at  the  Royal  Australian  Navy's  Mine  Warfare  Faculty  at  HMAS 
Waterhen,  Sydney.  The  day  involved  conducting  a  focus  group  discussion  session  with 
experienced  UUV  pilots.  Seven  Navy  UUV  pilots  (each  with  more  than  a  years  operational 
flight  experience)  attended  and  took  part  in  discussions  on  what  constituted  a  'typical' 
mission.  Upon  gaining  consent  from  all  involved,  the  session  was  tape  recorded  for  coding 
and  data  analysis  purposes  following  the  focus  group.  The  session  lasted  for  4  hours  and 
during  the  focus  group,  pilots  were  prompted  with  a  number  of  open  ended  questions 
(Appendix  B)  which  were  designed  to  generate  discussion  on  what  comprised  a  typical 
mission,  and  situation  awareness  information  requirements  that  were  considered  critical  to 
the  successful  completion  of  a  mission.  Discussions  assessed  relevant  goals  and  sub-goals 
associated  with  successfully  conducting  a  mission.  Situation  awareness  requirements  were 
then  determined  based  on  what  information  pilots  considered  both  essential  and  relevant 
to  be  able  to  conduct  their  task  and  achieve  the  overall  mission  goal.  The  proceeding 
interviews  involved  a  one-on-one  meeting  between  the  researcher  and  individual  UUV 
pilots  to  further  flesh  out  situation  awareness  requirements  as  related  to  display  design. 
The  interviews  which  lasted  30  minutes  each,  were  written  recorded  (upon  obtaining 
consent)  and  included  a  component  of  time  using  the  Navy's  UUV  pilot  training  simulator 
to  help  further  describe  and  demonstrate  information  requirements  related  to  achieving 
situation  awareness.  Due  to  time  constraints  placed  on  the  availability  of  UUV  pilots,  only 
4  of  the  7  pilots  were  interviewed. 

The  information  obtained  during  the  focus  group  sessions  and  one-on-one  interviews 
directly  contributed  to  the  conduct  of  a  GDTA  which  helped  generate  a  UUV  mission 
analysis  and  information  requirements  document  (Appendix  C).  The  GDTA  determined 
pilot  information  requirements  associated  with  achieving  and  maintaining  a  high  level  of 
situation  awareness  and  performance. 

The  content  of  the  focus  group  and  interviews  is  not  provided  in  full  in  the  current  report 
rather,  information  gleaned  from  the  session  relating  to  the  determination  of  generic  pilot 
situation  awareness  requirements  is  provided  in  Appendix  C.  The  three  displays  used  in 
this  experiment  were  developed  as  a  result  of  the  GDTA  conducted. 
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2.1.4  Experimental  Process 

In  order  to  compare  the  display  designs,  participants  were  asked  to  take  part  individually 
in  a  simulated  flight  experiment  in  which  three  different  scenarios  were  presented  to  each 
participant.  The  scenarios  were  designed  to  test  several  aspects  of  an  operator's  level  of 
situation  awareness  and  performance.  Each  of  the  three  scenarios  was  designed  to 
generate  data  that  would  allow  a  comparative  evaluation  of  the  UUV  operator  displays. 
Flight  data  from  the  simulator  were  recorded  in  a  log  for  each  participant  in  each  of  the 
three  scenarios.  The  data  collected  provided  information  on  the  operator's  level  of 
situation  awareness  and  performance.  For  each  scenario,  participants  performed  with  a 
different  display  design.  The  presentation  order  of  the  three  displays  was  randomised 
across  the  participants.  As  the  experiment  sought  to  determine  display  design  concepts 
which  promoted  SA  and  performance,  rather  than  being  an  experiment  specifically 
addressing  issues  of  navigation  and  task  difficulty,  the  presentation  of  the  flight  scenarios 
was  not  randomised  with  the  presentation  of  display  designs.  This  decision  was  justified 
on  the  grounds  that  each  of  the  three  flight  scenarios  were  of  equal  difficulty;  the  same 
terrain  was  used  for  each  scenario,  only  the  starting  point  varied. 

The  terrain  used  for  this  experiment  comprised  of  data  supplied  by  the  Navy's 
Hydrographic  Office.  The  coastal  region  was  selected  due  to  the  variability  in  underwater 
features  in  line  with  training  requirements  as  determined  by  the  focus  group  session  and 
interviews.  This  region  permits  pilots  to  operate  a  vehicle  in  full  six  degrees  of  freedom. 
The  region  selected  for  inclusion  in  the  scenarios  was  very  large  (1000m  x  1000m) 
however,  participants  did  not  utilise  the  full  area  rather  operated  in  a  small  section.  The 
same  section  of  terrain  was  used  for  all  three  scenarios,  only  the  starting  point  differed. 

A  number  of  waypoints  were  placed  at  various  locations  within  each  scenario.  Participants 
needed  to  utilise  the  instruments  and  information  provided  to  them  via  their  operator 
display  in  order  to  navigate  through  the  series  of  waypoints.  Each  of  the  operator  displays 
was  designed  to  help  pilots  gain  situation  awareness  and  enable  them  to  sufficiently 
manoeuvre  the  vehicle  towards  each  of  the  waypoints. 

All  participants  received  the  same  instructions  and  the  same  three  simulated  flight  tasks. 
None  of  the  test  group  was  familiar  with  any  aspect  of  the  simulator  prior  to  taking  part  in 
the  experiment.  Ethical  clearance  was  sought  and  granted  from  Monash  University's 
Standing  Committee  on  Ethics  in  Research  involving  Humans  (SCERH)  and  DSTO's 
internal  STAR  team.  As  an  ethical  requirement,  participants  received  an  explanatory 
statement  (Appendix  D)  to  read  and  a  consent  form  (Appendix  E)  to  sign. 

Participants  were  given  a  five  minute  session  to  familiarise  themselves  with  joystick 
controls  prior  to  commencement  of  the  experimental  conditions.  Participants  were 
provided  with  a  description  of  the  joystick  controls  (Appendix  A)  for  use  during  the 
practice  session.  They  were  instructed  to  read  through  the  instructions  and  become 
familiar  with  the  joystick's  movements.  Once  they  had  finished  reading  through  the 
instructions,  each  participant  was  given  a  5  minute  practice  session  with  the  simulator 
engaged  to  familiarise  themselves  with  the  joystick  and  vehicle  movements.  The 
simulation  program  was  loaded  and  participants  were  permitted  to  fly  around  within  the 
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simulated  environment  however,  during  this  5  minute  practice  session,  no  flight 
instruments  were  displayed. 

At  the  conclusion  of  the  5  minute  familiarisation  session,  participants  were  given  a 
description  of  the  task  they  were  asked  to  complete  (Appendix  F).  The  experimental  task 
required  participants  to  fly  to  a  set  of  waypoints  which  were  represented  on  a  chart 
located  in  the  top  right  hand  corner  of  their  operator  display.  Participants  used  the 
instruments  and  information  provided  to  them  on  their  operator  display  along  with  verbal 
instructions  from  the  researcher  in  order  to  manoeuvre  the  vehicle  towards  the  waypoints. 
Once  participants  had  read  the  task  description,  the  experiment  began.  The  researcher 
initiated  the  first  of  the  three  flight  tasks.  Using  the  simulation  environment,  participants 
were  asked  to  fly  a  mission  using  information  provided  to  them  both  verbally  by  the 
researcher,  and  visually  by  their  operator  display.  Participants  were  required  to  follow  an 
initial  heading  provided  by  the  researcher  in  order  to  progress  towards  each  of  the 
waypoints.  Participants  flew  towards  each  waypoint  verbally  acknowledging  to  the 
researcher  when  they  established  visual  contact  with  the  waypoint  as  seen  through  the 
vision  on  their  operator  display.  They  were  then  required  to  fly  through  the  waypoint. 
When  participants  acknowledged  visual  contact  with  the  waypoint,  the  researcher 
provided  a  relative  heading  to  the  next  waypoint.  Participants  were  instructed  to 
commence  along  that  new  heading  and  make  adjustments  where  necessary  to  reach  that 
waypoint.  The  vehicle  was  restricted  to  travel  below  the  water  surface  so  reaching 
waypoints  required  the  operator  to  navigate  around  and/or  over  underwater  objects  in 
order  to  reach  each  waypoint. 

2.1.5  Display  Design 

Display  designs  were  developed  using  principles  from  the  aviation  domain.  Two  of  the 
displays  were  based  on  the  Inside-Out  and  Outside-ln  design  principles.  The  Baseline 
display  that  was  used  (Figure  5)  is  analogous  to  the  display  currently  in  use  by  the  Royal 
Australian  Navy  for  the  Double  Eagle  MKll  Mine  Disposal  Vehicle.  The  second  display 
design  (Figure  6)  was  developed  based  on  applying  the  Inside-Out  design  principle  and 
the  third  display  (Figure  7)  was  developed  based  on  applying  the  Outside-ln  design 
principle.  Each  of  the  three  missions  presented  a  different  display  design  for  the  operator 
to  use. 
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Figure  5:  Baseline  display  design  presented  with  flight  task  1 
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Figure  6:  Inside-Out  display  design  presented  with  flight  task  2 


10 


DSTO-TR-1931 


Figure  7;  Outside-In  display  design  presented  with  flight  task  3 


The  three  display  designs  were  presented  to  participants  in  random  order  to  minimise  any 
systematic  effect  of  learning.  On  each  occasion,  the  three  displays  were  presented  with  the 
same  flight  task,  only  the  order  of  their  presentation  varied. 

The  chart  in  the  top  right  hand  corner  of  the  screen  was  used  as  a  navigational  aid  to 
indicate  when  a  participant  was  near  a  waypoint.  Waypoints  become  visible  on  the 
operator's  central  display  when  the  participant  was  within  200m  range  and  when  the 
circular  vehicle  indicator  on  the  reference  chart  overlaid  the  red  cross  which  depicted  the 
waypoints  (Figure  8). 
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Figure  8:  An  example  of  the  reference  chart  used  for  flight  task  3 

The  circular  vehicle  indicator  overlaid  the  red  waypoint  cross  as  participants  approached 
the  waypoint.  Participants  were  able  to  see  waypoints  which  were  represented  by  large 
orange  diamonds  within  each  scenario  (Figure  9),  upon  approach  if  the  circular  vehicle 
indicator  overlaid  the  red  cross  slightly.  Waypoints  were  not  necessarily  positioned  on  the 
sea  floor,  they  were  also  located  in  the  water  column  above  the  vehicle. 


Figure  9:  Example  of  a  waypoint  as  seen  visually,  and  on  the  reference  chart  in  each  scenario 
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2.1.6  Situation  Awareness  and  Human  Performance  Measures 

In  order  to  assess  a  participant's  level  of  situation  awareness,  at  random  intervals  during 
each  flight  task,  the  simulation  was  paused,  the  screen  was  blackened  and  participants 
were  prompted  by  a  series  of  questions  designed  to  assess  their  awareness  of  the  situation 
at  that  moment  (Table  1).  The  simulation  was  paused  a  total  of  eleven  times  for  each 
scenario  ensuring  all  questions  were  asked.  The  questions  asked  were  designed  to  probe 
all  levels  of  the  pilot's  situation  awareness.  Eleven  questions  were  developed  in  total  in 
accordance  with  information  obtained  in  the  GDTA.  Each  of  the  questions  was  presented 
in  random  order  and  at  random  times  during  each  display  design  condition.  The  timing  of 
the  presentation  of  the  questions  ensured  that  no  question  was  asked  before  80  seconds 
had  elapsed  in  each  scenario  and  even  then,  questions  were  presented  randomly  at 
intervals  of  between  60  seconds  and  75  seconds.  This  timing  ensured  that  all  questions 
would  be  asked  by  the  maximum  time  of  13  minutes  and  50  seconds  into  each  of  the 
scenarios.  Participants  were  permitted  to  fly  until  15  minutes  had  elapsed  at  which  point, 
the  scenarios  ended. 

The  questions  were  derived  from  the  GDTA  which  was  conducted  in  order  to  determine 
situation  awareness  requirements  for  operator  displays  (information  on  the  GDTA  and  the 
development  of  the  situation  awareness  probe  questions  is  provided  in  Appendix  C).  The 
timing  and  order  of  questions  was  randomised  for  each  experimental  scenario  to  avoid 
pre-empting  or  rehearsing  answers.  All  11  questions  were  asked  during  each  of  the  three 
conditions. 

Table  1:  SAGAT  Queries  for  UUV  remote  piloting  task 

1.  Estimate  your  current  pitch  angle  (in  degrees) 

2.  Estimate  your  current  roll  angle  (in  degrees) 

3.  Estimate  your  current  speed  (in  knots) 

4.  Estimate  your  current  depth  (in  metres) 

5.  Estimate  your  current  heading  (in  degrees  relative  to  north) 

6.  Estimate  the  number  of  the  next  waypoint  you  will  reach 

7.  Are  you  currently  accelerating,  decelerating  or  neither? 

8.  Which  direction  will  you  turn  in  the  next  10  seconds  (left,  right,  no  change?) 

9.  Estimate  the  distance  (in  metres)  to  the  next  waypoint  you  will  reach 

10.  Estimate  the  heading  required  to  reach  the  nearest  waypoint 

11.  What  mode  are  you  currently  in?  (Elight  or  Hover?) 


The  situation  awareness  probe  questions  related  to  the  instruments  and  information 
contained  on  the  operator  display  and  also,  where  participants  were  within  the 
environment.  In  question  11  relating  to  mode,  'Hover'  was  defined  as  allowing  the  vehicle 
to  maintain  altitude  and  position  about  or  over  a  place  or  object.  This  mode  was 
particularly  useful  when  approaching  a  waypoint.  The  vehicle  was  able  to  be  switched 
between  normal  Elight  mode  and  Hover  mode  which  allowed  it  to  manoeuvre  differently, 
hence  the  requirement  for  awareness  of  which  mode  it  was  in. 
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When  prompted  with  a  situation  awareness  question,  participants  were  required  to  voice 
their  answer  to  the  researcher  who  then  entered  it  into  the  question  prompt  box  on  their 
laptop  screen  (Figure  10).  Once  an  answer  was  logged,  the  scenario  continued. 

SA  was  measured  by  the  difference  between  the  correct  response  and  the  response 
provided  to  the  SAGAT  queries;  the  smaller  the  difference  between  responses,  the  higher 
the  level  of  SA  for  that  cue. 


Figure  10:  Example  of  Situation  Awareness  Question  asked  during  the  experimental  task 

Human  performance  measurements  were  also  taken  during  each  of  the  three  scenarios. 
The  simulator  logged  measurements  on  the  number  of  control  reversal  errors  participants 
made  in  each  scenario  with  respect  to  vehicle  roll;  the  number  of  waypoints  participants 
reached  during  each  scenario  and  also,  the  distance  they  travelled  during  each  scenario. 
These  measures  were  used  as  indicators  of  performance  with  respect  to  the  three 
competing  displays.  The  superior  display  design  was  measured  by  the  least  number  of 
control  reversal  errors  recorded,  the  greatest  number  of  waypoints  reached  and  also  the 
greatest  distance  travelled. 

Participants  were  given  15  minutes  to  fly  each  scenario.  Scenarios  came  to  an  end 
following  the  presentation  of  all  SA  questions  to  the  pilot.  There  was  a  short  rest  period  of 
one  minute  between  the  practice  session  and  each  of  the  experimental  scenarios.  The 
vehicle  was  initially  neutrally  positioned  in  terms  of  pitch  and  roll,  however  the  initial 
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depth  and  location  within  the  scenario  varied.  Participants  were  advised  to  keep  the 
vehicle  as  close  as  possible  to  between  10  -  20  metres  to  the  seabed  at  all  times.  At  the 
conclusion  of  all  three  flight  tasks,  participants  were  asked  if  they  had  a  preference  for  any 
of  the  displays  presented  and  were  asked  to  make  comments  on  each  of  the  three  displays 
presented. 
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3.  Results 


The  data  obtained  represent  responses  given  to  the  SA  probes  and  the  performance 
measurements  obtained.  The  independent  variable  involved  in  this  study  was  Display 
Design.  Statistical  analyses  were  performed  on  the  data  in  order  to  determine  any 
significant  relationship  between  Display  Design  and  the  dependent  variables  of:  level  of 
situation  awareness  and  human  performance. 

One-way  ANOVAs  were  performed  on  the  data  relating  to  seven  of  the  SA  probes  and 
also  the  performance  data.  An  ANOVA  was  conducted  for  each  of  probes  1-6  and  probe 
10  which  related  to;  pitch,  roll,  speed,  depth,  heading,  the  estimated  distance  to  the  next 
waypoint,  and  the  estimated  heading  to  the  next  waypoint  respectively.  SA  was  measured 
by  the  difference  between  the  correct  response  and  the  response  provided  to  the  SAGAT 
queries;  the  smaller  the  difference  between  responses,  the  higher  the  level  of  SA  for  that 
cue.  One-way  ANOVAs  were  conducted  on  the  performance  data  obtained  for:  the 
number  of  control  reversal  errors  recorded  for  vehicle  roll  in  the  Inside-Out  display 
condition,  the  number  of  waypoints  reached  in  each  condition,  the  odometer  reading  for 
each  condition.  A  speed  profile  was  also  generated  for  the  approach  to  the  first  waypoint 
in  each  condition.  An  ANOVA  was  conducted  on  the  data  to  determine  any  difference  in 
approach  speed  to  Waypoint  one.  Superior  performance  was  measured  by  the  least  number 
of  control  reversal  errors,  the  most  waypoints  reached,  the  greatest  odometer  reading  and 
the  display  with  the  highest  mean  speed  upon  approach  to  the  first  waypoint. 

No  correction  was  made  for  Type  1  error.  Each  measure  was  judged  to  be  largely 
independent  of  the  others  as  they  measured  different  aspects  of  performance. 

Based  on  the  results  obtained  for  all  participants,  comparison  was  made  between  the 
frequency  of  correct  and  incorrect  answers  given  to  SA  probes  7-9  and  probe  11  which 
related  to:  estimated  number  of  the  next  waypoint;  whether  the  vehicle  was  accelerating, 
decelerating  or  neither;  the  direction  the  vehicle  would  turn  in  the  next  10  seconds;  and  the 
vehicle's  current  mode  of  operation  (flight  or  hover).  Display  design  conditions  that 
yielded  the  highest  percentage  of  correct  answers  were  deemed  to  have  provided  higher 
levels  of  SA  for  that  cue.  Participants'  display  preference  was  also  recorded.  A  Chi-Square 
was  performed  on  the  data  to  determine  if  there  was  a  significant  preference  for  display 
design. 

3.1  Results  for  Situation  Awareness  Measures 

The  ANOVA  results  revealed  a  number  of  significant  outcomes. 

A  significant  effect  was  found  for  Display  Design  for  the  measure  of  vehicle  roll  (SA  query 
2)  F(2,  75)  =  4.277,  p  =  .018.  A  post-hoc  Tukey  HSD  test  revealed  a  significant  difference 
between  the  Baseline  and  Outside-ln  display  design  conditions  for  vehicle  roll  with  the 
Outside-ln  condition  providing  a  higher  level  of  SA  for  that  cue.  These  results  are  shown 
in  Figure  11. 
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Display  Design  Vs  Situation  Awareness  probe  -  Roll 
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Figure  11:  Mean  difference  between  display  design  for  SA  probe  2  relating  to  roll 

A  significant  effect  was  found  for  Display  Design  for  the  measure  of  vehicle  depth  (SA 
query  4)  F(2,  75)  =  3.617,  p  =  .032.  A  post-hoc  Tukey  HSD  test  revealed  a  significant 
difference  between  the  Inside-Out  and  Outside-ln  display  design  conditions  for  vehicle 
roll  with  the  Inside-Out  condition  providing  a  higher  level  of  SA  for  that  cue.  These  results 
are  shown  in  Figure  12. 
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Figure  12:  Mean  difference  between  display  design  for  SA  probe  4  relating  to  depth 
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No  significant  difference  was  observed  between  the  Display  Design  for  the  remaining  SA 
probe  measures  relating  to  pitch,  speed,  heading,  the  estimated  distance  to  the  next 
waypoint,  nor  the  estimated  heading  to  the  next  waypoint. 

Based  on  all  participant  responses,  a  comparison  was  made  between  the  percentage  of 
correct  responses  given  to  SA  probe  7:  Estimate  the  number  of  the  next  waypoint;  SA 
probe  8:  Are  you  accelerating,  decelerating  or  neither;  SA  probe  9:  Estimate  the  direction 
you  will  turn  in  the  next  10  seconds  and;  SA  probe  11:  What  mode  are  you  currently  in? 
(Elight  or  Hover?)  for  each  Display  Design.  The  Baseline  design  recorded  the  highest 
number  of  correct  responses  for  each  of  these  probes,  with  only  the  Inside-Out  condition 
sharing  an  equal  maximum  highest  percentage  of  100%  for  SA  probe  11.  These  results  are 
presented  in  Table  2. 


Table  2:  Percentage  of  correct  responses  across  participants  and  testing  times  for  SA  probes  7  to 
9  and  SA  probe  11  for  each  Display  Design 


Question 

n 

Baseline 

Inside-Out 

Outside-ln 

Q7.  Number  of  the  next  waypoint 

26 

76.9 

76 

65.4 

Q8.  Are  you  accelerating, 
decelerating  or  neither 

26 

84.6 

80 

80 

Q9.  Direction  you  will  turn  in  the 
next  10  seconds 

26 

88.5 

80 

73.1 

Qll.  What  mode  (Flight  or 
Hover)  are  you  currently  in 

26 

100 

100 

96.2 

3.2  Results  of  Performance  Data  Measurements 

A  significant  difference  was  observed  between  the  display  designs  for  the  number  of 
control  reversal  errors  observed  for  vehicle  roll  E(2,  76)  =  38.235,  p  =  .000,  shown  in  Eigure 
13.  A  post-hoc  Tukey  HSD  test  revealed  a  significant  difference  was  observed  between 
each  pair  of  the  Display  Designs  with  participants  having  the  least  number  of  control 
reversal  errors  for  in  the  Outside-ln  condition.  These  results  are  shown  in  Eigure  13. 
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Display  Design  vs  Number  of  Control  Reversal  Errors  for  Roll 


Display  Design 

Figure  13:  Number  of  control  reversal  errors  observed  vehicle  roll  for  each  display  design 

A  correlation  was  run  on  the  data  for  the  number  of  waypoints  reached  and  also,  the 
odometer  reading.  A  correlation  of  r  =  .775,  p=  .000  was  obtained  indicating  a  significant 
association  between  these  two  measures.  These  results  are  presented  in  Table  3. 


Table  3:  Correlation  between  the  Number  of  waypoints  reached  and  odometer  reading 


Correlations 

Number  of 
waypoints 
reached 

Odometer 

reading 

Number  of 

Pearson  Correlation 

1 

.775(**) 

waypoints  reached 

Sig.  (2-tailed) 

.000 

N 

77 

77 

Odometer  reading 

Pearson  Correlation 

.775(**) 

1 

Sig.  (2-tailed) 

.000 

N 

77 

77 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 

A  significant  difference  in  the  number  of  waypoints  reached  was  observed  between  the 
Display  Designs  F(2,  76)  =  17.371,  p  =  .000.  A  post-hoc  Tukey  HSD  test  revealed  a 
significant  difference  was  observed  between  each  pair  of  the  Display  Designs  with 
participants  reaching  the  most  waypoints  in  the  Outside-ln  condition  as  shown  in 
Figure  14. 
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Figure  14:  Number  of  waypoints  reached  for  each  display  design 

A  significant  difference  in  odometer  reading  scores  was  also  observed  between  the 
Display  Designs  F(2,  76)  =  4.483,  p  =  .015.  A  post-hoc  Tukey  HSD  test  revealed  that  a 
significant  difference  was  observed  between  the  Inside-Out  and  Outside-ln  conditions 
with  the  Outside-ln  condition  yielding  the  highest  odometer  reading.  These  results  are 
shown  in  Figure  15. 


Display  Design  vs  Odometer  reading 


Display  Design 


□  Odometer 
reading 


Figure  15:  Mean  Odometer  reading  for  each  display  design 
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A  speed  profile  was  generated  for  the  first  waypoint  in  each  Display  Design  condition. 
Waypoint  one  was  selected  for  inclusion  in  the  analysis  as  all  participants  reached  it  during 
each  of  the  three  Display  Design  conditions  presented.  As  shown  in  Figure  16,  the  speed 
profile  for  the  Baseline  and  Outside-ln  conditions  were  roughly  equivalent  while  the 
Inside-Out  condition  demonstrated  a  drop  in  performance  with  respect  to  the  other  two 
Display  Design  conditions.  A  significant  difference  in  approach  speed  to  the  first  waypoint 
was  observed  between  all  three  Display  Designs  F(2,  12)  =  7.215,  p  =  .009.  This  ANOVA 
was  run  on  the  value  obtained  for  the  mean  speed  of  all  participants  at  the  200m,  150m 
100m,  50m  and  Om  point  upon  approach  to  the  first  waypoint. 


Figure  16:  Average  speed  for  the  approach  to  Waypoint  one  for  each  display  design  condition 

A  Chi-Square  was  conducted  on  the  categorical  data  obtained  for  display  preference  and 
Age,  Gender  and  Experience.  Participants  showed  a  significant  difference  in  the  frequency 
for  preference  of  display  (2,  n  =26)  =  18.538,  p  =  .000  with  the  Outside-ln  display 
preferred  by  almost  75%  of  participants.  This  result  is  shown  in  Figure  17. 
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Figure  17:  Participants  preference  for  Display  Design 


In  addition,  two  one-way  ANOVAs  were  conducted  on  the  data  obtained  for  level  of 
experience  in  years  and  mode  of  experience  (in  terms  of  fixed  wing  aerial  pilot  or  UUV 
pilot)  against  SA  and  the  performance  measures  listed.  No  significant  difference  was 
observed  for  any  of  these  variables. 


22 


DSTO-TR-1931 


4.  Discussion 


The  results  contained  in  this  study  supported  the  hypothesis  that  Display  Design  would 
influence  a  pilot's  level  of  SA  with  the  Outside-ln  providing  for  comparatively  superior 
performance.  The  results  also  supported  the  hypothesis  that  Display  Design  would  affect 
pilot  performance  in  terms  of  the  number  of  control  reversal  errors  observed,  the  number 
of  waypoints  reached  and  the  total  distance  travelled  with  the  Outside-ln  design  again 
facilitating  superior  performance  over  the  remaining  displays. 

The  results  show  significant  differences  were  observed  between  the  display  types  in  terms 
of  level  of  situation  awareness  for  a  number  of  cues.  Significant  differences  were  observed 
between  designs  for  vehicle  roll  with  the  Outside-ln  condition  providing  a  higher  level  of 
SA  and  the  designs  for  vehicle  depth  with  the  Inside-Out  providing  a  higher  level  of  SA 
for  that  measure.  The  results  also  show  significant  differences  between  the  Display 
Designs;  in  the  number  of  control  reversal  errors  observed  for  roll  with  the  Outside-ln 
condition  producing  significantly  fewer  errors  than  the  other  two  displays;  the  number  of 
waypoints  reached  with  the  Outside-ln  condition  reaching  the  most  waypoints,  and  also 
the  odometer  reading  with  participants  travelling  further  in  the  Outside-ln  condition.  In 
accordance  with  these  findings,  results  also  indicate  the  Inside-Out  display  induced  worse 
approach  speed  to  Waypoint  one. 

Results  indicate  that  the  Outside-ln  concept  is  generally  suited  for  the  remote  control  of  a 
vehicle  for  a  number  of  reasons.  It  makes  sense  to  employ  an  Outside-ln  concept  for  these 
particular  displays  as  the  pilots  that  operate  such  vehicles  are  already  outside  of  the 
vehicle  (both  physically  and  psychologically)  and  the  Outside-ln  format  provides  the  best 
stimulus  response  compatibility  thus  reducing  the  incidence  of  control  errors. 

Interestingly,  the  results  obtained  for  this  experiment  both  support  and  contradict 
previous  research  findings  in  the  related  literature.  Whilst  in  general,  the  results  tended  to 
support  previous  findings  in  favour  of  an  Outside-ln  display  concept,  some  of  the  results 
for  the  SA  cues  and  performance  measures  were  contradictory  in  nature  to  commonly 
cited  conclusions  with  reference  to  the  use  of  Inside-Out  concepts. 

Analysis  of  the  data  collected  for  the  situation  awareness  measure  for  roll  provided  some 
interesting  results.  A  significant  difference  was  observed  between  Display  Design  for 
vehicle  roll  with  the  Outside-ln  representation  of  vehicle  roll  providing  the  highest  level  of 
situation  awareness  for  that  cue.  There  were  significantly  smaller  errors  in  roll  estimation 
for  the  Outside-ln  condition  over  the  Baseline  condition.  This  observation  may  be 
explained  by  the  reduced  visual  scanning  requirement  imposed  by  the  Outside-ln  display 
condition. 

Outside-ln  displays  can  be  considered  beneficial  in  that  they  reduce  the  need  for  visual 
scanning  between  panels  by  displaying  information  to  a  pilot  that  has  already  been 
integrated  -  such  as  information  about  vehicle  pitch  and  roll  [25].  This  reduces  the 
requirement  for  pilots  to  mentally  integrate  information  in  order  to  gain  an  understanding 
of  their  vehicle's  current  state. 
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In  the  Outside-In  condition,  information  on  vehicle  roll  is  located  centrally  on  the 
operator's  screen  providing  an  immediate  visual  representation  of  the  vehicle's  current 
roll  state.  This  information  is  integrated  with  information  on  vehicle  pitch  thus  requiring 
less  mental  processing  and  visual  scanning  on  the  part  of  the  operator.  The  results 
obtained  support  previous  research  findings  about  information  displayed  in  a  non- 
integrated  (separate  displays  or  co-planar)  format,  such  as  roll  and  pitch  are  in  the 
Baseline  display  [25].  Presenting  information  in  a  non-integrated  manner  adds  a  cognitive 
load  to  the  operator  as  attentional  resources  become  divided  between  scanning  and 
subsequent  processing  of  information.  This  increase  in  cognitive  load  results  from  the 
requirement  to  visually  scan  the  instruments,  then  process  and  integrate  information  in 
order  to  gain  an  accurate  understanding  of  the  vehicle's  current  state.  An  inability  to  gain 
an  accurate  understanding  of  a  vehicle's  state  within  its  current  operational  environment 
will  result  in  lower  levels  of  pilot  SA  which  has  the  potential  to  impact  on  performance 
thus  reducing  efficiency. 

The  findings  support  other  literature  which  indicates  that  contributions  of  the  scanning 
and  processing  requirements  of  non-integrated  (Baseline)  versus  integrated  Outside-In 
displays  will  impose  penalties  on  operator  performance  in  terms  of  accuracy  and  reaction 
time  [28,  29].  A  subsequent  integrated  Outside-In  representation  of  data  reduces  the 
cognitive  load  imposed  on  a  pilot  through  the  reduction  of  visual  scanning  and 
information  processing  thus  providing  an  observed  improvement  in  situation  awareness 
through  gaining  a  more  accurate  picture  of  the  vehicle's  current  state  [30]. 

Another  interesting  result  observed  for  SA  probe  2  relating  to  vehicle  roll  was  the  lack  of 
significant  difference  observed  between  the  Inside-Out  condition  for  roll  and  the  other  two 
Display  Designs  despite  the  very  high  incidence  of  control  reversal  errors  for  that 
particular  display. 

The  Inside-Out  (egocentric)  representation  of  pitch  and  roll  supports  the  principle  of 
pictorial  realism,  since  the  view  is  similar  to  what  is  seen  by  the  pilot.  However,  it  violates 
the  principle  of  the  moving  part  [5,  20].  For  example,  for  a  pilot  to  roll  the  vehicle  to  the 
left  would  require  a  leftward  joystick  movement  to  generate  a  counter-clockwise  rotation 
of  the  vehicle.  But  in  doing  so,  this  leftward  movement  of  the  control  input  produces  a 
clockwise  rotation  of  the  moving  element  on  the  operator's  display  -  the  artificial  horizon 
bar  -  in  a  way  which  is  counter-intuitive  to  the  pilot's  mental  model.  If  the  display  does  not 
respond  in  a  way  that  is  congruent  with  stick  input  and  the  pilot's  mental  model,  this 
increases  the  likelihood  of  control  reversal  errors  -  a  reduction  in  the  stimulus-response 
compatibility  [22,  23,  31].  In  this  study,  the  Outside-In  depiction  of  roll  and  corresponding 
control  input  does  not  produce  as  many  control  reversal  errors  as  the  Inside-Out  condition 
as  matching  the  control  motion  with  the  motion  of  the  display  preserves  the  inherent 
stimulus-response  compatibility  relationship  [32,  33].  These  results  can  be  compared  to 
similar  findings  by  Worringham  &  Berringer  (1998)  who  looked  at  control  response  as 
related  to  an  operator's  visual  field.  They  concluded  that  the  field  of  view  and  directional 
stimulus-response  compatibility  are  intimately  linked.  Fewer  control  reversal  errors  were 
observed  when  the  control  motion  is  in  the  same  direction  as  the  display  motion  [34]. 
Similarly,  other  researchers  have  found  a  reduction  in  control  reversal  errors  in  cases 
where  display  motion  is  compatible  and  congruent  with  the  frame  of  reference  of  the 
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control  input  [25,  35-37].  While  there  was  no  significant  difference  observed  for  roll 
between  the  Outside-ln  and  Inside-Out  conditions,  in  light  of  the  high  number  of  control 
reversal  errors  observed  for  the  Inside-Out  display,  adopting  the  use  of  the  Outside-ln 
display  for  roll  proves  most  appropriate. 

In  addition  to  producing  more  control  reversal  errors,  it  was  interesting  to  note  that  the 
Inside-Out  condition  did  not  perform  as  well  on  other  performance  measures  as  its  design 
counterparts.  The  Inside-Out  performed  the  least  well  out  of  the  three  designs  on  the  other 
performance  measures  in  terms  of  the  number  of  waypoints  reached,  the  final  odometer 
reading  and  also  the  speed  profile  for  approach  to  Waypoint  one.  These  findings  are  not  in 
accordance  with  literature  in  the  aviation  domain  which  found  a  significant  effect  of 
Display  Design  and  the  mean  time  to  traverse  all  waypoints.  In  a  study  conducted  by 
Wickens  et  al.  [29],  the  Inside-Out  display  format  revealed  a  clear  and  predicted  advantage 
for  guidance  tasks.  This  result  was  not  replicated  in  the  current  study  with  the  Inside-Out 
condition  performing  significantly  worse  that  the  other  displays  on  the  number  of 
waypoints  reached.  This  reduction  in  performance  may  be  due  to  the  information 
processing  costs  associated  with  this  display  design.  Higher  processing  requirements  on  a 
task  generally  result  in  a  speed  vs.  accuracy  trade-off,  the  generalisation  is  that  when 
speed  is  of  the  essence,  accuracy  will  decrease  and  vice  versa,  hence  there  is  a  trade-off  in 
functioning  between  them.  This  may  explain  the  reduced  performance  in  that  pilots  were 
devoting  so  much  attention  to  processing  information  that  they  sacrificed  speed  for 
accuracy.  This  would  explain  why  they  reached  significantly  fewer  waypoints  and  did  not 
travel  as  far  on  average. 

The  stimulus  response  compatibility  of  the  displays  has  already  been  cited  as  an  issue 
which  would  have  the  ability  to  affect  overall  performance.  For  participants  to  travel  a 
shorter  distance  as  recorded  by  their  odometer  and  thus,  reach  fewer  waypoints  during 
each  mission  indicates  an  uncertainty  or  that  they  were  less  comfortable  with  the  display. 
The  speed  profile  as  observed  for  each  of  the  three  Display  Design  conditions  indicated 
that  the  approach  speed  pattern  to  Waypoint  one  for  the  Baseline  and  Outside-ln  designs 
was  roughly  equivalent.  Pilots  appeared  to  be  able  to  maintain  a  constant  and  higher 
speed  for  longer  on  approach  to  the  waypoint  which  indicated  they  were  comfortable  with 
workings  of  the  display.  The  speed  profile  observed  for  the  Inside-Out  condition  showed  a 
significant  reduction  in  performance  in  terms  of  being  able  to  maintain  speed  at  a  constant 
and  high  level.  As  a  consequence  of  this,  participants  obviously  tended  to  reach  fewer 
waypoints  and  thus,  had  lower  final  odometer  readings  than  the  other  displays.  This 
result  can  not  be  considered  a  function  of  the  difficulty  of  the  task  as  complexity  was 
matched  overall  for  all  three  conditions.  This  observed  reduction  in  performance  for  the 
Inside-Out  display  has  particular  implications  for  mission  effectiveness.  If  the  display  does 
not  foster  improved  performance,  then  mission  efficiency  will  be  affected.  In  reviewing  the 
results  obtained  for  the  performance  measures  for  the  Inside-Out  display,  a  clear  reduction 
in  overall  performance  is  observed  thus  indicating  this  type  of  display  is  less  suitable  for 
enhancing  performance  for  UUV  vehicle  piloting  purposes  than  the  Baseline  and  Outside- 
ln  displays. 

The  results  obtained  for  the  situation  awareness  measure  of  depth  were  interesting.  The 
Inside-Out  depiction  of  a  depth  counter  produced  the  highest  level  of  situation  awareness 
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than  the  other  two  Display  Designs.  The  Inside-Out  depth  indicator  was  a  fixed  pointer- 
moving  scale  design,  similar  to  that  which  has  been  adopted  for  most  aircraft  displays 
relating  to  altitude  [38]  while  in  the  other  displays,  depth  was  indicated  by  a  fixed  scale- 
moving  pointer  representation.  The  results  obtained  in  the  present  study  support  the 
application  of  the  fixed  pointer  -  moving  scale  design  as  a  suitable  way  to  represent  height 
above  or  below  a  given  point,  as  seen  with  their  use  in  the  aviation  environment. 
However,  a  disadvantage  of  such  displays  is  that  digital  values  become  hard  to  read  when 
the  variable  is  changing  rapidly  since  the  digits  themselves  are  moving.  In  the  aviation 
environment,  this  becomes  an  issue  due  to  the  speeds  and  ease  of  movement  aircraft 
achieve  however,  its  effects  become  less  of  an  issue  when  considering  use  of  this  type  of 
display  in  the  underwater  environment  due  to  the  reduced  speed  at  which  these  vehicles 
operate.  Coupled  with  the  reduction  in  speed  execution  in  the  operational  environment, 
this  particular  type  of  display  may  also  prove  superior  as  it  reduces  the  requirement  for 
visual  scanning.  The  fixed  pointer  is  readily  locatable  within  the  operator  screen  unlike  the 
Outside-In  depiction  of  depth  in  which  the  value  scrolls  up  and  down  the  operators  screen 
depending  on  their  relative  depth  thus  increasing  visual  scanning  requirements. 

The  results  obtained  for  the  situation  awareness  questions  7-9  and  11  relating  to  the 
number  of  the  next  waypoint;  whether  the  vehicle  was  accelerating,  decelerating  or 
neither;  the  direction  the  vehicle  would  turn  in  the  next  10  seconds  and;  what  mode  the 
vehicle  was  currently  in  respectively,  indicate  that  the  Baseline  display  was  superior  in 
terms  of  providing  the  highest  percent  accuracy  for  those  cues.  While  the  other  displays 
tended  to  fare  slightly  worse,  this  may  be  due  to  the  information  processing  requirements 
associated  with  them.  Also,  the  fact  that  performance  on  the  Outside-In  display  was  higher 
than  the  Baseline  in  terms  of  number  of  waypoints  reached,  it  is  reasonable  to  assume  that 
the  further  a  pilot  gets  into  the  task,  the  less  likely  they  will  be  to  remember  the  number  of 
the  next  waypoint  making  that  particular  cue  (for  that  condition)  a  memory  task  rather 
than  a  performance  task.  In  short,  the  more  waypoints  a  pilot  reached,  the  less  likely  they 
were  to  recall  how  many  they  had  passed  through. 

In  addition  to  the  above  analyses,  comparison  was  made  on  a  number  of  categorical 
variables  to  determine  if  there  were  any  additional  influences  which  could  have  explained 
a  reduction  or  improvement  in  performance.  In  contrast  to  previous  findings  where 
training  novice  pilots  to  master  an  Inside-Out  display  took  much  longer  than  for  the 
moving  aircraft  display  [13,  39],  the  results  obtained  did  not  support  this.  In  terms  of  level 
of  experience  and  mode  of  experience,  the  results  obtained  revealed  no  difference  between 
Display  Design  for  the  level  of  performance  on  either  the  SA  or  the  performance  ratings. 
Novice  and  experienced  pilots  (both  UUV  and  fixed  wing  pilots)  performed  equally  well 
on  each  of  the  display  designs.  There  was  also  no  difference  observed  between  Display 
Design  and  performance  for  age  or  gender.  In  spite  of  the  measurable  outcomes, 
participants  had  a  clear  preference  for  the  Outside-In  display  over  the  other  two  display 
conditions. 

This  study  provided  a  number  of  interesting  results  which  have  implications  for  both 
theory  and  practice  when  considering  display  design  for  remotely  piloted  vehicles, 
however  a  number  of  limitations  were  imposed  on  the  current  study.  While  the  results  of 
this  study  are  generally  encouraging,  a  number  of  additional  variables  could  have 
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influenced  the  results  obtained.  While  some  participants  were  UUV  and  fixed  wing  pilots, 
the  majority  of  the  participants  were  flight  naive  participants,  some  of  which  had  never 
operated  a  joystick  before.  Accounting  for  outliers  in  the  data,  this  still  may  have  impacted 
on  results.  While  flight  experience  did  not  appear  to  have  an  effect  on  the  level  of  SA  or 
performance,  a  certain  level  of  knowledge  of  display  motion  and  vehicle  control 
mechanisms  ought  to  be  assumed  due  to  the  nature  and  complexity  of  performing  such 
tasks.  In  hindsight,  it  might  have  been  advantageous  to  recruit  only  flight  experienced 
pilots  to  participate  however,  this  would  have  proved  very  difficult  to  arrange.  A  follow- 
on  study  aims  to  further  investigate  pilot  training  and  experience  in  greater  detail  in  order 
to  determine  any  influence  of  experience  with  respect  to  performance  on  tasks. 

The  current  study  provided  results  which  have  implications  for  the  wider  military 
community  when  considering  display  design  for  remotely  piloted  vehicles.  A  variety  of 
control  displays  are  currently  being  used  for  these  vehicles,  many  with  deficiencies  [14] 
and  research  aimed  at  improving  unmanned  vehicle  control-station  design  is  underway  in 
a  number  of  laboratories.  The  findings  demonstrate  that  not  all  the  principles  of  display 
design  employed  in  the  aviation  domain  are  readily  applicable  to  the  underwater 
environment  and  conclusions  drawn  from  the  current  study  will  assist  researchers  in  this 
area  in  determining  a  suitable  display  design  which  produces  superior  performance  on  all 
variables. 

Whilst  this  study  did  replicate  a  number  of  results  gleaned  from  the  aviation  literature, 
some  were  in  contrast  indicating  the  apparent  differences  in  these  operating  environments. 
Noted  differences,  such  as  degrees  of  freedom  of  vehicle  movement,  and  the  speed  of  task 
execution  in  each  of  the  operating  environments  indicate  that  assuming  universality 
between  the  two  operating  environments  is  not  advisable,  rather  research  for  the 
underwater  environment  should  be  conducted  in  its  own  right  to  experimentally 
determine  the  nature  and  relative  significance  of  these  differences.  In  line  with  this,  results 
indicate  improvement  in  UUV  operator  situation  awareness  and  performance  can  be 
achieved  by  improving  the  design  of  the  current  operator  display.  Significant  differences 
were  observed  between  display  designs  and  the  level  of  situation  awareness  and 
performance,  these  differences  were  observed  for;  vehicle  roll,  depth,  number  of 
waypoints  reached,  control  reversal  errors  observed  for  roll  in  the  Inside-Out  condition 
and  the  speed  profile  for  approach  to  waypoints.  As  a  consequence  of  these  findings,  a 
'composite'  display  is  proposed  which  encompasses  all  attributes  of  the  displays  that 
produced  the  best  performance  and  provided  higher  levels  of  situation  awareness  (Figure 
18).  The  composite  display  comprises  of  the  Outside-ln  depiction  of  roll  &  pitch,  the 
Baseline  depiction  of  heading  and  the  Inside-Out  depiction  of  depth.  In  the  case  of  the 
representation  of  vehicle  heading  where  statistical  significance  was  not  achieved,  the 
attribute  representing  vehicle  heading  from  the  Baseline  (Mine  Disposal  Vehicle)  display 
was  retained.  While  this  format  for  displaying  heading  was  not  altered,  it  is  considered  an 
Outside-ln  display. 
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Figure  18:  'Composite'  display  for  enhanced  situation  awareness  and  performance 


The  current  findings  are  encouraging  for  work  on  display  design  for  remotely  piloted 
vehicles.  As  this  study  is  one  of  the  first  human  factors  studies  conducted  for  UUVs,  many 
future  research  endeavours  will  follow.  A  necessary  next  step  is  to  investigate  the 
effectiveness  of  the  composite  display  in  its  entirety  in  terms  of  its  provision  of  enhanced 
situation  awareness  and  performance.  To  confirm  the  results  obtained  in  the  present 
study,  the  composite  display  should  be  tested  against  the  current  Baseline  display  to 
determine  overall  effectiveness.  Implementation  of  the  composite  display  in  the  real 
operating  environment  will  also  be  a  necessary  step  to  ensure  its  suitability.  In  addition  to 
this,  the  control  mechanisms  through  which  remotely  piloted  underwater  vehicles  are 
operated  need  to  be  investigated.  Given  the  complexity  of  the  environment,  and  the  ability 
of  the  vehicle  to  move  in  six  degrees  of  freedom,  controlling  the  vehicle  can  be  very 
difficult.  It  seems  appropriate  to  investigate  the  vehicle's  current  manual  control 
mechanisms  in  order  to  determine  their  suitability  for  aiding  superior  performance  in 
conjunction  with  the  use  of  the  proposed  composite  display  (Figure  18).  Additionally,  pilot 
training  programs  and  pilot  selection  criteria  should  be  investigated  to  see  if  these 
variables  have  any  influence  on  mission  effectiveness. 
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Human  factors  research  for  UUVs  is  a  new  and  exciting  domain  which  presents  many 
research  opportunities.  As  indicated,  many  issues  exist  that  can  combine  to  influence 
overall  system  performance,  particularly  in  the  case  of  a  remotely  piloted  vehicle.  When 
seeking  to  improve  overall  system  performance,  display  design  becomes  a  very  important 
issue  as  it  has  the  ability  to  influence  situation  awareness  and  performance.  In  summary, 
the  results  of  the  current  study  tended  to  support  similar  research  findings  in  the  aviation 
domain.  However,  some  discrepancies  between  the  operating  environments  are  noted, 
such  as  the  degrees  of  freedom  of  vehicle  movement  and  the  speed  of  task  execution 
contrasting  the  two  environments.  Results  indicate  situation  awareness  and  pilot 
performance  can  be  improved  through  the  investigation  of  Display  Design,  a  result  which 
has  implications  for  human  factors  studies  into  the  operation  of  unmanned  vehicles  right 
around  the  world. 
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5.  Literature  Review 


Over  recent  years.  Unmanned  Vehicles  (UVs)  have  emerged  as  a  sensible  and  practical 
alternative  to  human  involvement  in  many  situations.  As  such,  these  vehicles  have  been 
embraced  by  commercial  and  military  establishments  around  the  world.  Increasingly,  UVs 
of  all  forms  are  taking  on  roles,  particularly  in  the  military  environment,  where  the  risk  to 
human  life  is  considered  too  high  or  where  environmental  conditions  are  too  inhospitable 
for  humans  [40]. 

In  today's  technological  environment,  many  different  types  of  transport  UVs  exist  and  are 
utilised  for  a  wide  range  of  purposes.  By  far  the  most  prominent  in  this  family  are  the 
unmanned  air  and  ground  vehicles  and  as  such,  much  of  the  research  into  UVs  has 
focused  specifically  on  these  arms  of  the  family  with  interests  mainly  confined  to  system 
technologies  [9-12].  An  emerging  maritime  equivalent  also  exists,  the  Unmanned 
Underwater  Vehicle  (UUV)  which  now  plays  an  important  role  in  underwater  operations. 
While  the  UUV  is  recognised  as  part  of  this  family,  they  have  not  been  subject  to  the  same 
level  of  research  as  their  air  and  land  counterparts.  To  that  end,  the  UUV  -  or  Remotely 
Operated  Vehicle  (ROV)  as  they  are  also  known  -  can  somewhat  be  considered  a  new  class 
of  socio-technical  system. 

The  human  element  plays  a  significant  part  in  any  socio-technical  system.  Events  such  as 
the  disasters  that  occurred  at  Chernobyl,  Bhopal  and  Three  Mile  Island  reveal  the  potential 
catastrophic  nature  of  failure  at  the  man-machine  interface.  Further  to  this,  the  NASA 
Challenger  space  craft  accident  showed  us  even  space  is  not  immune  from  human-designed 
technological  mistakes.  Although  safety  is  constantly  improving  in  some  technologies,  the 
concept  of  Human  System  Integration  (HSl)  with  respect  to  a  socio-technical  system  is 
largely  ignored,  or  only  adequately  engineered  following  review  of  a  demonstrated 
mishap  [41].  By  focusing  early  on  the  human  element  in  the  design,  testing  and 
implementation  process,  it  is  possible  to  achieve  dramatic  increases  in  system  performance 
and  overall  productivity  [41].  In  order  to  achieve  this.  Human  Factors  (HF)  issues  require 
critical  attention  during  the  development  of  any  functional  system  -  particularly  an 
unmanned  system.  When  employing  a  UV,  and  seeking  to  combine  the  remoteness  of  the 
environment  with  some  form  of  human  intelligence,  the  ultimate  goal  is  that  via  the  use  of 
sensors,  the  operator  will  be  able  to  act  and  perform  as  if  he  or  she  was  really  present  at 
the  remote  location  [15].  To  achieve  this  end,  consideration  of  HF  will  remain  essential  to 
the  design,  development  and  implementation  of  an  unmanned  vehicle,  thus  contributing 
to  overall  system  growth  and  maturity. 

As  a  field  of  research,  the  exploration  of  human  factors  for  UUVs,  both  military  and 
commercial,  is  in  its  early  stages.  To  date,  relatively  few  UUV  HF  studies  have  been 
published  in  the  open  literature,  a  reflection  of  the  infancy  of  this  domain  of  research. 
UUVs  constitute  an  area  of  growing  interest  due  to  their  ability  to  operate  at  depths  and  in 
areas  that  are  inaccessible  to  humans  or  other  types  of  vessels.  These  vehicles  are  used 
extensively  for  underwater  search  and  salvage,  inspection,  surveying,  scientific 
exploration,  and  mine  countermeasures.  Owing  to  their  operational  capabilities,  the  Royal 
Australian  Navy  (RAN)  employs  UUVs  to  complete  a  wide  range  of  tasks,  including  mine 
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counter-measure  applications  and  hydrographic  surveillance  operations.  Due  to  the 
important  roles  these  vehicles  assume  in  such  situations,  it  is  necessary  to  ensure  superior 
systems  and  technologies  are  in  place  in  order  to  facilitate  optimal  performance  -  by  both 
human  and  machine. 

Of  the  open  literature,  the  majority  of  HF  studies  relating  to  UUVs  primarily  focus  on  pilot 
training  issues  and  techniques  [42-44].  These  studies  explore  training  methodologies  with 
the  intent  to  examine  and  quantify  the  transfer  effect  of  learned  skills  to  live  operations. 
Due  to  the  expense  and  logistics  involved  with  operating  an  actual  UUV,  training  and 
practice  is  often  difficult  to  obtain  other  than  whilst  "on  the  job".  In  an  attempt  to  address 
this,  Fletcher  and  Harris  (1996)  have  developed  and  demonstrated  a  virtual  environment 
based  system  (TRANSoM)  for  training  ROV  piloting  skills.  Simulation  based  technology 
can  be  an  effective  tool  in  pilot  training  and  results  to  date  have  yielded  comparable 
piloting  performance  between  the  TRANSoM  system  and  an  actual  ROV  [42,  43] . 

While  simulation-based  technology  demonstrates  effectiveness  in  the  training  arena,  its 
use  has  not  been  extended  to  consider  other  human  factors  issues  that  pertain  to  UUV 
operations.  While  training  will  undoubtedly  remain  a  focus  for  HF  researchers,  significant 
attention  needs  to  be  directed  towards  other  features  of  the  piloting  experience  which  also 
have  the  ability  to  influence  Human  Performance  (HP)  and  mission  operations.  Research 
into  workload  with  respect  to  differing  levels  of  automation,  human  control  processes, 
training  aspects  and  importantly,  issues  surrounding  Situation  Awareness  (SA)  for  UUV 
operators  are  all  very  important  topics  which  need  to  be  fully  examined  [40]. 

Situation  Awareness  is  a  very  important  HF  issue  when  considering  UUV  operations  as  it 
has  the  potential  to  greatly  influence  the  success  of  operations.  SA  is  a  prominent  HF  issue 
as  it  influences  performance  in  light  of  the  tasks  pilots  must  perform  as  part  of  the  system. 

Situation  Awareness  as  a  concept  can  be  defined  as  "the  perception  of  elements  in  the 
environment  within  a  volume  of  time  and  space,  comprehension  of  their  meaning  and 
projection  of  their  status  into  the  near  future"  [17].  Being  'aware'  of  what  is  happening 
around  you  and  understanding  what  information  means  to  you  now,  and  in  the  future,  is 
critical  to  the  success  of  a  task  in  complex  settings.  As  the  concept  of  SA  usually  applies  to 
operational  situations  (where  SA  is  required  for  a  specific  reason)  it  can  usually  be  defined 
in  terms  of  what  information  is  important  for  a  particular  job  or  goal. 

SA  has  been  shown  to  be  a  critical  element  in  the  success  of  a  mission  [19].  If  persons 
within  the  system  of  interest  do  not  possess  adequate  SA,  the  performance  of  the  system  as 
a  whole  will  be  degraded.  For  this  reason,  SA  remains  an  ongoing  consideration  during 
system  operation,  particularly  for  the  pilots  of  UUVs. 

Endsley  (1995a,  1995b)  developed  a  theoretical  model  of  SA  in  order  to  provide  an 
understanding  of  the  processes  and  factors  that  influence  the  development  of  SA  in 
complex  settings.  The  model  (illustrated  in  Figure  19)  shows  the  overall  role  of  SA  in 
decision-making  and  performance.  While  SA  is  a  stage  physically  separate  from  decision¬ 
making  and  performance  of  actions,  it  is  highly  influential  in  decision-making  and  as  such, 
can  be  considered  the  main  precursor  to  this  process.  Based  on  that  premise,  operators 
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derive  information  from  their  environment  in  order  to  decide  what  to  do  about  a  situation, 
or  to  carry  out  any  necessary  actions  or  tasks.  Intuitively,  decisions  about  specific 
situations  are  made  based  on  an  operator's  level  of  SA.  If  an  operator  does  not  possess 
adequate  SA  for  a  situation,  it  is  reasonable  to  assume  that  faulty  or  ill-informed  decisions 
could  result  as  a  by-product  of  this  reduced  SA. 


Figure  19:  Endsley's  model  of  situation  awareness  in  dynamic  decision  making  [17] 


As  a  concept,  situation  awareness  is  an  inherent  element  of  what  is  defined  under  the 
broader  heading  of  human  performance.  Human  Performance  refers  to  the  level  or 
accuracy  and  efficiency  with  which  an  operator  completes  a  given  task  [45] .  If  an  operator 
has  a  high  level  of  SA,  then  it  is  feasible  to  suggest  that  overall  HP  within  the  system  may 
also  be  high.  In  many  socio-technical  systems,  different  aspects  of  human  performance  can 
be  measured  directly  from  system  data  output.  An  operator's  level  of  SA  can  be 
determined  through  a  measure  of  SA  and  correlated  against  actual  system  data  to  get  a 
feel  for  how  SA  affects  different  aspects  of  performance  such  as  time  on  task  and  accuracy. 

Many  elements  can  combine  negatively  to  influence  an  individual's  level  of  SA  and 
performance,  however,  these  influences  can  largely  be  alleviated  if  they  are  duly 
considered  during  the  system  design  phase.  The  effectiveness  of  system  design  appears  to 
have  a  major  influence  on  the  development  of  a  high  level  of  SA  during  operational 
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situations  [27].  In  order  to  establish  the  degree  to  which  new  technologies  and  design 
concepts  improve  or  degrade  operator  SA,  evaluation  of  these  concepts  must  take  place 
using  measures  of  SA  to  determine  potential  positive  and  negative  consequences  of  design 
characteristics. 

As  SA  plays  an  important  role  in  the  success  of  operations,  it  is  therefore  necessary  to 
examine  things  that  influence  it  with  the  aim  of  improving  and  maintaining  appropriate 
levels  of  SA.  For  many  systems,  operator  SA,  workload  and  other  performance  measures 
can  be  directly  measured  during  design  concept  testing.  Specifically,  if  SA  is  measured 
directly,  it  should  be  possible  to  select  design  concepts  that  promote  SA  and  thus,  increase 
the  probability  that  operators  will  make  effective  decisions  and  avoid  poor  ones  [27]. 
Many  of  the  problems  associated  with  SA  such  as  non-integrated  data,  automation  issues, 
and  excess  attention  demands,  are  frequently  brought  on  by  poorly  thought  out  system 
design.  These  factors  can  be  detected  early  in  the  design  process  where  corrective  changes 
can  be  made  to  improve  the  design  [19]. 

Current  research  into  situation  awareness  has  focused  on  its  measurement  in  dynamic 
systems  where  efforts  have  focused  on  the  aviation  domain  [18].  This  has  been  borne  out 
of  the  significant  role  that  aircraft  (manned  and  unmanned)  continue  to  play  within  the 
civil  and  military  arena.  However,  this  is  not  to  imply  that  these  techniques  should  be 
confined  solely  to  SA  research  in  the  air  domain,  rather  the  techniques  available  are  also 
applicable  to  other  domains  and  as  such,  can  be  applied  to  HF  research  into  UUVs. 

SA  measurement,  and  the  evaluation  of  potential  design  concepts,  routinely  takes  place 
within  the  context  of  rapid  prototyping  or  human-in-the-loop  task  simulation  [19]. 
Although  the  gathering  of  SA  knowledge  can  be  achieved  in  the  real  time  environment, 
measurement  within  a  simulated  environment  provides  researchers  with  greater  flexibility 
to  evaluate  design  concepts  allowing  the  design  process  the  iterations  it  needs  in  order  to 
develop  a  system  which  promotes  and  helps  maintain  operator  SA. 

With  the  advent  of  simulation  based  technology,  many  of  the  problems  associated  with 
situation  awareness  and  human  performance  can  be  alleviated  through  the  system  design 
and  evaluation  phase.  The  ultimate  goal  of  any  socio-technical  system  is  to  provide  the 
correct  environment  and  tools  for  the  human  operator  to  complete  their  intended  task 
with  a  high  degree  of  efficiency  and  a  minimum  of  error.  Poorly  designed  systems 
exacerbate  error  and  decrease  overall  efficiency,  however,  these  costly  effects  need  not 
occur.  Consideration  and  integration  of  situation  awareness  and  human  performance 
requirements  should  be  mandatory  during  the  development  process  of  any  system. 

5.1  Information  Requirements:  Display  Design  Issues 

A  UUV  has  no  onboard  pilot  and  as  a  result,  the  success  of  a  mission  relies  on  proficient 
human  operators  on  the  ground  or  surface  platform  to  ensure  appropriate  guidance  for 
the  task  at  hand.  However,  having  correct  and  relevant  information  to  aid  completion  of 
the  task,  remains  highly  influential  in  ensuring  the  success  of  a  mission.  If  a  pilot  is  not 
provided  with  adequate  information,  and  in  a  manner  which  is  easily  accessible  and 
decodable,  then  performance  and  efficiency  will  suffer [16].  It  is  therefore  necessary  to 
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explore  the  tools  UUV  pilots  rely  on  to  aid  safe  and  effective  completion  a  mission. 
Undoubtedly,  the  Human  Computer  Interface  (HCl)  as  presented  to  pilots  via  the  means 
of  their  operator  display  (essentially  a  Head-Up  Display  (HUD))  [7],  must  be  of  sufficient 
calibre  and  relevance  for  them  to  perform  the  task  to  the  best  of  their  ability.  It  is  for  this 
reason  that  pilot  information  displays  should  be  investigated  to  ensure  pilots  are  being 
presented  with  information  that  is  necessary,  correct  and  relevant  to  the  operational  task. 

Display  design  has  a  huge  impact  on  the  operation  of  many  socio-technical  systems  and 
many  human  interface  challenges  have  arisen  in  the  UUV/ROV  domain,  each  of  which 
has  the  potential  to  contribute  to  a  reduced  SA,  therefore  leading  to  degraded  mission 
performance  [45].  Currently,  UUV  information  display  design  has  been  based  upon  the 
same  principles  and  implemented  in  the  same  way  as  for  those  unmanned  vehicles 
operating  in  the  air  domain  [13].  However,  as  UUVs  operate  in  a  completely  different 
environment  from  that  of  an  Unmanned  Aerial  Vehicle,  it  is  not  sufficient  to  merely 
employ  the  same  principles  as  those  explored  for  the  air  domain  in  UUV  operations. 
Certain  features  of  the  underwater  operating  environment  such  as  sea  state,  water  current, 
salinity,  temperature  profiles,  and  turbidity  have  the  ability  to  adversely  influence  an 
operational  mission.  In  addition,  one  notable  difference  between  the  two  operating 
environments  is  the  speed  at  which  the  vehicles  are  operated.  In  the  aviation  domain, 
vehicles  are  operated  in  a  very  fast  environment  which  allows  swift  and  decisive  vehicle 
movements.  Consequently,  display  design  becomes  an  issue,  as  within  this  real  time 
environment  displays  may  become  difficult  to  read  with  rapid  information  changes.  This 
increase  in  speed  of  execution  and  display  of  information,  places  demand  on  the  operator 
in  terms  of  information  processing  and  reaction  time  when  making  decisions.  Thus  SA  can 
be  differentially  affected  if  information  is  not  displayed  in  an  easily  interpretable  and 
coherent  fashion.  Rate  of  change  of  display  information  becomes  less  of  an  issue  when 
considering  display  design  in  the  underwater  environment  due  to  the  reduced  speed  at 
which  these  vehicles  operate.  A  reduction  in  speed  execution  in  the  operational 
environment  means  information  displayed  to  pilots  is  presented  at  a  less  rapid  rate  which 
has  the  potential  to  reduce  information  processing  times.  A  drawback  of  this  is  that  as 
things  happen  in  a  less  rapid  environment,  tasks  take  longer  to  execute  which  may 
hamper  the  maintenance  of  pilot  SA  due  to  the  volume  of  information  they  must  keep 
track  of. 

Display  design  is  of  high  importance  for  both  air  and  underwater  vehicles  due  to  the 
integral  part  it  plays  in  allowing  a  pilot  to  execute  a  mission  successfully.  The  speed  of 
execution  of  the  operating  environment  contrasts  a  major  difference  between  the  operation 
of  these  two  class  of  vehicles  however,  as  mentioned  it  is  not  the  only  distinguishing 
feature.  As  such,  due  to  the  variation  between  the  two  operating  environments,  it  is 
possible  that  different  elements  may  affect  operating  performance  at  both  the  human  and 
the  machine  level.  It  is  necessary  therefore  that  HF  research  for  UUVs  be  conducted  in  its 
own  right  in  order  to  confirm  or  disconfirm  any  potential  universality  between  the  classes 
of  vehicle. 

In  light  of  the  infancy  of  the  domain  of  research,  it  seems  intuitive  to  seek  to  examine  the 
current  HCl  at  the  level  of  display  design  and  information  presentation  in  order  to  answer 
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some  very  important  initial  HF  issues.  In  reviewing  the  related  literature,  it  becomes 
apparent  that  display  design  is  an  important  issue  which  warrants  further  investigation. 

5.2  Current  and  Explored  Display  Issues 

A  number  of  studies  have  been  conducted  in  the  air  and  land  domains  which  have  sought 
to  address  critical  issues  with  respect  to  pilot  information  displays  [1-4,  7] .  Resulting  from 
this  research,  a  number  of  principles  have  been  identified  on  what  constitutes  a  "good"  or 
"bad"  design;  unfortunately  however,  very  few  of  these  principles  have  been  validated  in 
models  that  could  help  to  identify  the  "best"  design,  or  determine  how  principles  trade-off 
against  each  other  [46].  Furthermore,  issues  such  as  display  separation,  task  demand  and 
display  clutter  [1]  have  been  investigated  with  the  intention  of  determining  the 
characteristics  of  a  high-quality  display  in  terms  of  content  and  layout  that  optimises  pilot 
performance  [46]. 

Further  to  this,  researchers  have  explored  3D  perspective,  and  multi-sensory  displays  to 
improve  operator  telepresence,  operator  SA  and  mission  performance  [40,  47-50].  Bemis, 
Leeds  and  Winer  (1988)  compared  performance  on  a  conventional  plan-view  (top  down) 
display  with  performance  on  a  perspective  display.  Subjects  in  their  experiment  were 
required  to  detect  a  threat  and  select  the  closest  interceptor  to  deal  with  the  threat.  Bemis 
et  al.  found  a  significant  reduction  in  errors  of  detection  and  interception  with  the  use  of  a 
perspective  display,  as  well  as  a  reduced  response  time  for  selection  of  interceptors  [51].  In 
a  study  investigating  air  traffic  avoidance  tasks,  Ellis,  McGreevy,  and  Hitchcock  (1987) 
found  subjects  took  less  time  to  select  avoidance  manoeuvres  and  were  more  likely  to 
achieve  separation  when  using  a  perspective  display  than  a  conventional  view  display. 
Naikar,  Skinner,  Leung  and  Pearce  (2001)  conducted  a  similar  experiment  where  operator 
performance  on  a  conventional  two-dimensional  (2D)  tactical  display  was  compared  with 
performance  on  a  three-dimensional  (3D)  perspective  display  [52].  In  concurrence  with 
Bemis  et  al.  (1988),  results  indicated  an  advantage  for  the  perspective  display  over  the  2D 
displays,  although  this  was  somewhat  dependent  on  the  conditions  being  tested. 

While  these  researchers  concluded  that  perspective  displays  showed  potential  to  reduce 
errors  and  response  time  for  operators,  the  evidence  to  date  on  this  issue  has  not  been 
consistent.  Other  researchers  [53]  have  found  that  top-down  plan  view  displays  to  be  as 
good,  if  not  better,  than  perspective  displays  when  precise  distance  judgments  were 
required.  Perspective  displays  can  be  ambiguous  regarding  precise  position  and  can 
introduce  distortions  in  perceived  locations  in  the  attempt  to  depict  depth  [25,  54].  Bearing 
these  findings  in  mind,  the  issue  of  distortions  in  perceived  location  is  a  critical  one  which 
is  particularly  important  when  considering  whether  to  present  information  in  a  top  down 
plan  view  or  a  perspective  view  to  a  UUV  pilot.  As  operators  can  be  provided  with  a 
wealth  of  information  relating  to  their  vehicle,  how  this  information  is  displayed,  and  in 
what  format  becomes  important.  Any  distortion  in  perceived  location  is  a  critical  issue, 
particularly  if  the  distortion  is  a  result  of  a  poor  display  design.  Although  these  findings 
appear  to  lend  general  support  to  the  use  of  a  perspective  display  in  the  real  time  piloting 
of  an  aircraft,  any  distortion  would  have  an  impact  on  a  pilot's  SA  and  therefore,  should 
be  investigated.  While  these  findings  relate  to  aircraft,  they  must  also  be  considered  when 
addressing  the  HF  issues  associated  with  operating  a  UUV.  An  inaccurate  understanding 
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of  the  vehicle's  location  within  their  environment  becomes  particularly  important, 
especially  during  mine  counter-measure  operations. 

In  addition  to  providing  an  operator  with  accurate  and  easily  interpretable  information  via 
their  operator  display,  telepresence  becomes  a  significant  requirement.  Telepresence  refers 
to  the  perception  of  presence  within  a  physically  remote  or  simulated  site  -  the  ability  to 
project  an  operator's  perceptual,  cognitive  and  psychomotor  capabilities  into  distant, 
dangerous  or  simulated  environments.  Telepresence  has  been  identified  as  a  design  ideal 
for  synthetic  environments  [55] . 

Telepresence  and  Situation  Awareness  are  two  concepts  which  should  go  hand  in  hand. 
While  the  role  of  human  operators  in  teleoperated  systems  depends  largely  on  the 
system's  level  of  automation,  achieving  appropriate  telepresence  will  remain  an  important 
factor  in  the  operation  of  a  UUV.  Situation  Awareness  can  be  considered  a  good  indicator 
of  operator  presence  within  a  virtual  world,  or  when  conducting  remote  operations  of  a 
vehicle  such  as  an  unmanned  air  or  underwater  vehicle. 

So  how  do  we  achieve  adequate  telepresence?  What  type  of  display  promotes  telepresence 
and  enhances  an  operators  situation  awareness  and  human  performance?  The  literature 
suggests  that  3D  perspective  displays  have  a  general  advantage  over  their  2D  counterparts 
in  providing  more  global  awareness  [8]  however,  this  finding  is  not  universal  [30] . 

In  reviewing  the  literature  on  display  design,  it  becomes  apparent  that  there  are  costs  and 
benefits  to  presenting  data  in  both  a  2D  and  3D  format.  Consequently,  the  effectiveness  of 
technologies  such  as  a  UUV  will  depend  on  the  particular  display  options  chosen  and  the 
context  within  which  the  technology  is  applied. 

5.3  Head-Up  Displays 

One  widely  accepted  way  to  provide  information  to  pilots  is  through  the  use  of  a  Head-Up 
Display  (HUD)  which  integrates  information  from  sensors  and  the  outside  world  into  a 
working  picture  of  the  operating  environment.  The  concept  of  a  HUD  is  not  novel;  in  fact, 
the  world's  first  flight  worthy  ground-referenced  HUD  was  developed  in  the  1960's  by  the 
Australian  Aeronautical  Research  Laboratory  (formerly  ARL,  now  DSTO).  These  displays 
are  now  used  in  virtually  every  combat  aircraft  in  the  world.  The  advantages  of  a  Head- 
Up  Display  over  conventional  head-down  displays  which  require  the  pilot  to  break  visual 
continuity  with  their  outside  world  have  been  observed  [56-58],  and  the  positive  impact 
HUDs  have  had  on  flying  has  been  well  documented  [59,  60] .  A  number  of  benefits  have 
been  associated  with  flying  with  a  HUD,  including  improved  aircraft  flight  control, 
particularly  in  areas  such  as  instrument  approaches  in  inclement  weather  [57].  HUDs  have 
the  benefit  of  lessening  pilot  workload  while  presenting  the  information  in  a  position  that 
should  keep  them  well  informed  of  the  systems  operation  [38].  However,  the  benefits 
associated  with  the  implementation  of  a  HUD  do  not  mean  that  they  are  infallible,  as  it  is 
conceivable  that  there  may  be  situations  when  the  benefits  of  HUD  use  are  minimal, 
nonexistent,  or  possibly  reversed  [46]. 
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While  there  is  general  consensus  that  HUDs  are  useful  in  presenting  information  to  a  pilot, 
how  this  information  is  displayed  can  have  a  marked  impact  on  a  pilot's  performance. 
HUDs  can  display  information  to  pilots  in  a  variety  of  ways  from  graphical  format,  to 
straight  numerical  output,  to  an  egocentric  vs.  exocentric  view  of  flight  data.  In  a  detailed 
examination  by  Johnson  &  Roscoe  [20]  the  issue  of  how  to  display  information  to  pilots  is 
explored  as  it  is  noted  that  pilot  performance  can  be  influenced  by  display  design  and 
information  content.  When  considering  a  pilot's  frame  of  reference  with  respect  to  how 
they  view  their  vehicle;  either  as  egocentric  (vehicle  referenced),  or  as  an  exocentric  (world 
referenced)  frame  of  reference,  this  has  the  potential  to  impact  on  a  pilot's  ability  to 
achieve  and  maintain  SA.  This  in  turn  will  influence  performance  and  as  a  consequence, 
reduce  the  effectiveness  of  a  mission.  In  short,  the  manner  in  which  information  is 
displayed,  particularly  to  a  UUV  pilot,  becomes  a  very  important  issue. 

While  studies  have  been  conducted  to  demonstrate  the  usefulness  of  HUDs  in  automobiles 
and  aircraft  [61]  being  operated  by  people  in  the  'live'  environment,  little  has  been  done  to 
test  the  effectiveness  of  Head-Up  information  displays  for  remotely  piloted  vehicles. 
Drawing  on  research  conducted  for  HCI  in  the  live  world,  examination  of  display  design 
and  information  content  as  it  applies  to  unmanned  underwater  vehicles  is  suggested  as  a 
priority  HF  research  agenda.  It  is  conceivable  that  display  design  and  information  content 
for  UUV  operational  interfaces  will  differ  significantly  from  that  of  a  real  time  aircraft 
display  and  as  such,  operators  of  a  UUV  may  require  extra  information,  such  as  tactile 
feedback  of  their  vehicle  in  the  operational  environment  for  example. 

Human  performance,  specifically  human  control  processes  and  situation  awareness  are 
two  prominent  HF  issues  that  need  to  be  addressed  with  respect  to  UUV  operations. 
Necessarily  ingrained  in  the  process  of  display  design,  HP  &  SA  play  a  very  important  role 
in  the  development  and  evaluation  of  new  systems  and  interface  designs. 

Due  to  the  relative  infancy  of  the  UUV  HF  research  field,  a  necessary  step  is  to  evaluate  the 
suitability  of  existing  UUV  interface  designs  with  the  intent  to  determine  how  they  aid  and 
promote  superior  system  performance.  A  good  display  design  is  an  undoubtedly 
necessary  tool  to  help  promote  and  maintain  a  high  level  of  operator  performance  and 
situation  awareness.  Display  design  should  be  viewed  therefore  as  a  top  priority  when 
considering  the  human  factors  issues  associated  with  operating  an  Unmanned  Underwater 
Vehicle. 

The  aim  of  the  present  study  is  to  extend  on  current  work  conducted  for  display  design  in 
the  aviation  domain  and  apply  it  to  the  operation  of  UUVs  in  order  to  help  determine  any 
universality  between  the  underwater  and  air  domains. 
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Appendix  A:  -  Joystick  movements 

The  joystick  operates  in  two  modes.  These  modes  allow  for  the  full  6  degrees  of  freedom 
(movement)  of  the  vehicle.  The  two  modes  to  be  aware  of  are; 

•  Flight  mode; 

•  Hover  mode. 

The  default  mode  is  Flight  mode.  The  vehicle  will  start  each  scenario  in  Flight  mode.  In 
this  mode,  the  vehicle  acts  much  like  an  aircraft.  The  joystick  controls  for  Flight  mode  are 
as  follows: 


•  Acceleration:  Controlled  by  the  throttle  lever  at  the  bottom  centre  of  the 
joystick  base.  The  throttle  must  be  set  to  the  full  off  position  (pointing  towards 
the  minus  sign)  at  the  commencement  of  each  scenario.  Moving  the  throttle 
forwards  and  backwards  will  control  acceleration  and  deceleration  of  the 
vehicle. 

•  Pitch  Control:  Pushing  the  joystick  straight  forward  and  pulling  the  joystick 
straight  backward  in  Flight  mode  will  control  the  pitch  of  the  vehicle.  When  the 
joystick  is  pushed  or  pulled  and  allowed  to  return  to  the  neutral  position,  the 
vehicle  will  maintain  the  pitch  set  by  the  operator. 

•  Roll  Control:  Pushing  and  holding  the  joystick  left  to  right  in  Flight  mode  will 
control  the  roll  of  the  vehicle.  When  the  joystick  is  moved  to  the  left  or  right 
and  allowed  to  return  to  the  neutral  position,  the  vehicle  will  maintain  the  roll 
set  by  the  operator. 

•  Yaw  Control:  Twisting  the  joystick  from  left  to  right  will  control  the  yaw 
(heading)  of  the  vehicle.  When  the  joystick  is  twisted  left  or  right  and  allowed 
to  return  to  the  neutral  position,  the  vehicle  will  maintain  the  yaw  (heading)  set 
by  the  operator. 

•  Button  11:  Button  11  is  located  to  the  left  of  the  joystick.  Button  11  will  switch 
the  vehicle  to  Reverse  Flight  to  enable  you  to  fly  backwards  if  required.  Please 
note  to  activate  Reverse  Flight,  the  throttle  must  be  returned  to  the  full  off 
position.  The  current  direction  of  flight  is  displayed  in  the  bottom  left  hand 
corner  of  the  operator  screen  in  each  scenario. 

•  Button  12:  Button  12  is  also  located  to  the  left  of  the  joystick.  Button  12  will 
switch  the  vehicle  back  to  Forward  Flight  from  Reverse  Flight  if  required.  The 
current  direction  of  flight  is  displayed  in  the  bottom  left  hand  corner  of  the 
operator  screen  in  each  scenario. 

There  are  two  ways  to  switch  the  vehicle  controls  between  Flight  mode  and  Hover  mode. 
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Acknowledgement  of  the  current  mode  is  displayed  in  the  bottom  left  hand  corner  of  the 
operator  screen  in  each  scenario. 

Modes  can  be  selected  and  maintained  by; 

•  Selecting  between  Flight  and  Hover  mode:  Clicking  the  'hat'  on  top  of  the 
joystick  forwards  and  backwards  will  select  between  flight  and  hover  mode. 
Once  clicked  in  either  direction,  the  vehicle  will  remain  in  that  mode  until  the 
operator  selects  another  mode. 

•  Temporary  mode  selection:  The  vehicle  can  be  temporarily  switched  between 
each  of  the  modes  by  pulling  and  holding  the  trigger  located  at  the  position  of 
the  forefinger  on  the  underside  of  the  joystick. 

There  is  no  restriction  to  what  mode  the  operator  may  fly  the  vehicle  in.  He  or  she  may 
choose  to  fly  in  a  combination  of  Flight  and  Hover  mode  for  greater  control.  The  joystick 
controls  for  Hover  mode  are  as  follows: 

•  Acceleration:  Controlled  by  the  throttle  lever  along  the  bottom  centre  of  the 
joystick  base.  The  throttle  must  be  set  to  the  full  off  position  (pointing  towards 
the  minus  sign)  at  the  commencement  of  each  scenario.  Moving  the  throttle 
forwards  and  backwards  will  control  acceleration  and  deceleration  of  the 
vehicle. 

•  Depth  Control:  Pushing  the  joystick  straight  forward  and  pulling  the  joystick 
straight  backward  in  Hover  mode  will  control  the  depth  of  the  vehicle.  When 
the  joystick  is  pushed  or  pulled  and  allowed  to  return  to  the  neutral  position, 
the  vehicle  will  maintain  the  depth  set  by  the  operator. 

•  Horizontal  Displacement:  Pushing  and  holding  the  joystick  left  to  right  in 
Hover  mode  will  control  the  sideways  movement  (horizontal  displacement)  of 
the  vehicle.  When  the  joystick  is  moved  to  the  left  or  right  and  allowed  to 
return  to  the  neutral  position,  the  vehicle  will  maintain  the  position  set  by  the 
operator. 

•  Yaw  Control:  Twisting  the  joystick  from  left  to  right  will  control  the  yaw 
(heading)  of  the  vehicle.  When  the  joystick  is  twisted  left  or  right  and  allowed 
to  return  to  the  neutral  position,  the  vehicle  will  maintain  the  yaw  (heading)  set 
by  the  operator. 

You  are  encouraged  to  familiarise  yourself  with  the  joystick  controls  and  engage  the 
vehicle  in  all  degrees  of  freedom  during  your  familiarisation  session,  and  also  during  the 
experimental  scenarios. 

If  you  have  any  questions  about  the  control  of  the  vehicle  using  the  joystick,  please  ask  the 
researcher  during  your  5  minute  control  familiarisation  session.  These  joystick  control 
instructions  will  not  be  provided  for  reference  during  the  actual  scenarios. 
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Appendix  B:  HMAS  Waterhen  Visit:  Plan  for 
information  collection 


As  part  of  MPD's  Human  Factors  investigations  for  Unmanned  Underwater  Vehicles,  a 
focus  group  session  has  been  arranged  in  order  to  interview  operators  of  the  Double  Eagle 
mine  hunting  UUV.  The  focus  group  will  take  place  on  Thursday  August  5*  at  HMAS 
Waterhen. 

The  session  will  be  conducted  to  help  gather  information  on  UUV  operator  activities.  This 
information  will  be  compiled  and  used  as  accompanying  analyses  for  a  DSTO  based 
experimental  program  which  will  investigate  the  effect  of  display  design  on  UUV  operator 
Situation  Awareness  (SA)  and  human  performance. 

B.l.  Outline  of  information  being  sought 

As  mentioned,  the  focus  group  session  will  take  place  in  order  to  gather  information  on 
operator  activities  and  information  requirements  for  the  task  of  flying  an  MDV.  With  the 
consent  of  the  participants,  the  session  will  be  tape  recorded  and  written  records  will  be 
taken  in  order  to  gather  generic  and  detailed  data. 

Operators  will  be  asked  to  talk  about  the  current  system  in  order  to  gain  a  general  idea  of 
what  each  operator  does  in  the  course  a  mission  and  any  problems  they  encounter. 

The  session  will  have  a  broad  focus  and  will  cover  a  range  of  topics  designed  to  collect  as 
much  information  as  possible.  Broadly,  information  sought  will  be  related  to  the 
following: 


•  The  nature  of  a  Typical  Mission 

•  A  typical  operational  environment 

•  Essential  information  to  complete  piloting  task 

•  Possible  improvements  to  the  current  system  (e.g.  control  mechanisms,  HCl 
upgrade  etc) 

Part  of  the  session  will  consist  of  stepping  through  a  typical  mission  with  a  view  to 
identifying  the  moment-by-moment  task  demands  on  the  MDV  operator.  The  information 
required  by  the  operator  to  meet  the  task  demands  will  also  be  identified,  together  with 
the  available  information  sources. 

At  the  end  of  the  session,  a  prototype  situation  awareness  display  screen  will  be  presented 
for  comment.  The  display  has  been  designed  to  be  applicable  across  the  widest  possible 
range  of  underwater  remotely  operated  vehicle  types  and  missions,  and  frank  discussion 
of  its  merits  with  respect  to  MDV  operations  will  be  welcomed. 
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B.2.  Goal  Directed  Task  Analyses 

In  order  to  test  pilot  skills  and  knowledge  requirements  for  the  task  at  hand,  the  session 
will  also  include  discussion  of  the  major  goals  and  subgoals  that  are  to  be  achieved  during 
a  typical  mission.  A  Goal  Directed  Task  Analyses  (GDTA)  will  be  conducted  subsequent  to 
the  session  and  will  be  based  on  the  information  provided.  The  GDTA  will  obviously 
examine  the  goals  of  the  mission  (e.g  to  detect/ identify/ classify  an  object)  and  the  steps 
that  lead  up  to,  and  tasks  that  need  to  be  completed,  in  order  to  achieve  overall  operational 
goals. 

The  information  contained  in  the  GDTA  will  be  used  directly  to  determine  the  Situation 
Awareness  (SA)  requirements  of  the  operators.  Subsequent  SA  probe  questions  will  be 
developed  to  be  asked  during  the  experimental  flying  sessions.  The  SA  questions  will  be 
developed  around  the  sort  of  information  typically  displayed  to  the  pilots  (in  the 
experimental  simulation,  via  the  Head  Up  Display  HUD).  Pilots  will  be  able  to  answer  the 
questions  based  on  how  well  they  use  they  use  the  information  contained  in  the  HUDs 
during  the  piloting  experiment. 

B.3.  Any  Stand  Out  Issues? 

The  interview  group  will  be  asked  to  give  their  thoughts  on  any  issues  they  feel  impede 
their  ability  to  complete  their  tasks  and  achieve  their  goals.  Broad  topics  likely  to  be 
covered  include: 

•  Discussion  on  physical  controls  for  the  vehicle,  e.g.  do  the  pilots  consider  the 
current  two  joystick  setup  to  be  sufficient?  Have  they  thought  about  the 
possibility  of  one  joystick  controller  for  ease  of  use?  Do  they  utilise  the  step 
functions  for  fine  controlling  more  than  using  the  joysticks? 

•  Issues  associated  with  cable  management? 

•  Training  issues?  What  does  the  current  training  consist  of?  Is  the  current  pilot 
training  adequate?  If  not,  what  would  they  change? 

If  anyone  has  any  questions  or  concerns  in  relation  to  the  session,  please  contact  either 
Sarah-Louise  Donovan  (DSTO)  on  (03)  9626  8618 

( S  arah-Louise .  Dono van@dsto .  defenc  e .  go v .  au)  or  Peter  Henley  (Curtain  University)  on  (08) 
9266  3555  (pihenlev@iinet.net.au). 
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Constraining  facts  of  flight 

•  Vehicle 

•  Obstacles  in  environment 

-  Visibility 

-Turbidity  currents 

-Terrain 
-  Flight  rules 

Sitiiatioii  Awareness  Queries  I  I  I  I  I  I  I  I  I  I  I  I 

1  1  1  1  1  1  1  1  1  1  1  1  1  1 

Situaticin  Awareness  queries  qenerated  from  GDTA  (Fxus  qrouptranserpt  not  included  due  to  contents). 

SA  queries  have  been  derived  from  the  Focus  qrouo  session  and  reflect  SA  requirementsfor  UUV  oilotinq  tasks 

1  1  1  1 

SA  Level 

01 

Estimate  vour  current  oitch  ancle  fin  decreesl 

Ll/2 

02 

Estimate  vour  current  roll  ancle  fin  deareesl 

L1/2 

03 

Estimate  vour  current  sceed  fin  knots') 

Ll/2 

04 

Estimate  vour  current  deoth  (in  metresf 

Ll/2 

06 

Estimate  vour  current  headina  fin  decrees  relatve  to  north) 

Ll/2 

06 

Estimate  the  distance  fin  metres)  to  the  next  wavoont  vou  will  reach? 

L2 

07 

Estimate  the  number  of  the  next  vwivcoint  vou  will  reach? 

L3 

08 

Are  vou  currentiv  acceleratinq .  dec^eratinc  or  neither?  1 

L3 

09 

Which  direction  will  vou  turn  in  the  next  10  seconds?  fleft  rieft  no 

hance?) 

13 

O10 

Estimate  headne  reouired  to  reach  the  nearest  wavooirt 

12 

Oil 

What  mode  are  vou  currently  in?  fFliqht  or  Hover?)  I 

Ll/2 

Situation  Awareness  Requirements  Minehunting 

-  Depth/Altltude 

-  Heading 

-  Pitch 

-Ron 

-  Position  reiative  to  vessel  (exocentric  view) 

-  Position  relative  to  other  significant  objects 
(exocentric  &  egocentric  view  points) 

-  Speed  and  direction  through  water 

-  Speed  and  direction  over  seabed 

-  Time  (to  target,  to  complete  etc) 

-  visual _ 


Figure  1.  Usa' Litci  fiicc  (lesiaupi  ocess  as  cited  inEiidslcy.2003  (^I  R.  Eiidslcy.  Bolte. 
B.&Joties.D.G..2003). 


This  Goal  Directed  Task  Analysis  (GDTA)  was  conducted  based  on  the  outcomes  from 
a  number  of  Focus  Group  sessions  with  experienced  UUV  pilots  at  the  Navy's  Mine 
Warfare  Faculty.  Due  to  the  nature  of  the  information  provided  in  the  Focus  Group 
sessions,  the  official  transcripts  can  not  be  provided  with  the  GDTA.  However,  the 
transcripts  directly  contributed  to  the  determination  of  Situation  Awareness  requirements 
for  UUV  operations  (Minehunting).  The  Situation  Awareness  requirements  as  outlined 
above  formed  the  basis  of  information  to  be  displayed  on  each  of  the  screens  provided 
in  the  experimental  conditions.  The  same  information  was  presented  in  each  of  the 
three  conditions,  only  the  format  of  presentation  differed.  The  Situation  Awareness 
queries  that  were  developed  were  a  result  of  the  GDTA  and  were  designed  to  assess 
participants  level  of  Situation  Awareness  with  respect  to  the  displays  presented.  For 
a  more  detailed  description  of  the  development  of  this  model  to  represent  the  GDTA. 
please  contact  the  author. 


A  model  of  the  procedure  that  was  undertaken.  First  establishment  of  the  SA  requirements. 
2nd  involved  the  design  of  suitable  displays  based  on  I/O  &  0/1  principals.  Thirdly, 
measurement  in  the  form  of  test  and  evaluation  (Simulation  environment  utilising 
SAGAT  and  performance  measures)  to  determine  final  design.  This  is  an 
iterative  process  which  is  highly  valuable  when  determining  display  design 
requirements  that  will  enhance  performance  and  SA. 
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Appendix  D:  Explanatory  Statement 

D.l.  Project  Title:  Investigating  the  Effects  of  Display  Design  on 
Unmanned  Underwater  Vehicle  Pilot  Performance. 

My  name  is  Sarah-Louise  Donovan  and  I  am  doing  research  under  the  supervision  of  Prof. 
Tom  Triggs  a  Professor  in  the  Department  of  Psychology  towards  a  PostGrad  Dip  in 
Psychology  at  Monash  University. 

The  aim  of  this  research  is  to  investigate  the  effect  of  different  user  interface  designs  on  the 
performance  of  an  Unmanned  Underwater  Vehicle  (UUV)  operator/pilot.  The  experiment 
will  involve  presenting  participants  with  three  competing  displays  implemented  within  a 
simulated  environment.  Using  the  simulation  environment,  participants  will  be  asked  to 
navigate  a  path  using  information  provided  to  them;  verbally  by  the  researcher,  and 
visually  by  their  Head-Up  operator  display.  Measurements  of  human  performance  (speed, 
time,  accuracy)  and  Situation  Awareness  will  be  gathered  in  order  to  help  establish  if  a 
particular  display  type  or  configuration  provides  superior  performance  among  UUV 
operators/ pilots.  The  experiment  will  go  for  approximately  1  -  1.5  hours  duration. 

Data  from  the  experiment  will  be  collected  in  two  forms  being;  human  performance 
measurements  (for  speed,  time  taken  and  accuracy)  will  be  collected  by  the  simulation 
program  during  the  experimental  conditions  and  secondly;  measurements  will  be  taken  on 
operator  Situation  Awareness  throughout  the  experiment.  Situation  Awareness  data  will 
be  collected  electronically  by  a  series  of  probe  questions  asked  about  each  experimental 
condition.  The  simulation  will  be  paused  at  random  intervals  and  Situation  Awareness 
questions  will  be  presented  requiring  a  response  for  the  mission  to  continue. 

The  outcome  of  this  experiment  has  implications  for  both  commercial  and  military  use  of 
UUVs  and  as  such,  the  results  of  this  study  will  be  made  available  to  relevant  Defence 
personnel  and  also,  will  possibly  be  published  in  relevant  publications. 

Each  participant  will  be  asked  to  sign  a  consent  form  indicating  their  voluntary 
participation  in  the  experiment.  Names  will  not  be  recorded  for  the  experiment  and  data 
collection  instead  participants  will  only  be  required  to  provide  the  researcher  with  their 
date  of  birth  and  gender.  Access  to  data  collected  will  be  restricted  to  myself  as  researcher 
and  the  project  supervisor.  When  the  project  is  completed  and  written  up,  each  participant 
will  receive  a  copy  of  the  results  to  acknowledge  their  contribution  to  the  process.  All 
written  reports  and  data  will  be  stored  for  at  least  5  years  as  prescribed  by  the  university 
and  DSTO  regulations. 

The  risk  of  physical  and  psychological  stress  associated  with  this  project  is  judged  to  be 
minimal.  The  nature  of  the  experiment  and  data  collection  is  designed  to  be  minimally 
intrusive  and  should  pose  little  inconvenience  or  discomfort  to  participants.  Participants 
are  merely  asked  to  participate  in  a  simulated  'computer  game  like'  flying  exercise  where 
their  knowledge  of  the  situation  and  their  performance  will  be  recorded  and  compared 
within  the  bounds  of  the  experiment. 
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You  may  withdraw  from  the  experiment  at  any  time  simply  by  informing  myself  or 
another  researcher  involved  in  the  project.  You  will  not  be  required  to  give  a  reason  either 
to  myself  or  any  other  researchers,  and  neither  not  participating  at  all  nor  withdrawing 
will  have  a  negative  effect  on  your  employment  position  or  promotion  prospects. 

If  you  have  any  queries  regarding  this  project  or  would  like  to  be  informed  of  the 
aggregate  research  findings,  please  contact  telephone  the  researcher  on  (03)  9626  8618  or 
fax  (03)  9626  8652. 


You  can  complain  about  the  study  if  you  don't  like  something  about  it.  To  complain  about  the 
study,  you  need  to  phone  9905  2052.  You  can  then  ask  to  speak  to  the  secretary  of  the 
Human  Ethics  Committee  and  tell  him  or  her  that  the  number  of  the  project  is  2003/809. 
You  could  also  write  to  the  secretary.  That  person's  address  is: 

The  Secretary 

The  Standing  Committee  on  Ethics  in  Research  Involving  Humans 
PO  Box  No  3A 
Monash  University 
Victoria  3800 

Telephone  +61  3  9905  2052  Fax  +61  3  9905  1420 
Email:  SCERH@adm.monash.edu.au 


Thank  you. 


Sarah-Louise  Donovan 
(03)  9626  8618 
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Appendix  E:  Consent  Form 


Project  Title:  Investigating  the  effects  of  display  design  on  Unmanned  Underwater  Vehicle 

pilot  performance 

I . agree  to  take  part  in  the  above  Monash  University/ DSTO 

research  project.  I  have  had  the  project  explained  to  me,  and  I  have  read  the  Explanatory 
Statement  which  includes  information  on  how  to  contact  the  researchers,  which  I  keep  for  my 
records.  I  understand  that  agreeing  to  take  part  means  that  I  am  willing  to: 

•  Take  part  in  a  simulated  flight  experiment 

•  Allow  data  from  my  participation  to  be  recorded  and  analysed  for  each  experimental 
condition 

•  Be  interviewed  by  the  researcher  in  order  to  determine  my  level  of  Situation 
Awareness  for  the  experimental  conditions  I  am  participating  in 

•  Make  myself  available  for  a  further  interviewing  and  experimental  condition  testing 
should  that  be  required 

As  outlined  in  the  Explanatory  Statement,  I  understand  the  purposes  for  which  the 
information  is  being  collected.  I  understand  and  acknowledge  that  the  risk  of  physical  and 
psychological  stress  associated  with  this  project  is  judged  to  be  minimal.  The  nature  of  the 
experiment  and  data  collection  is  designed  to  be  minimally  intrusive  and  should  pose  little 
inconvenience  or  discomfort  to  participants.  Should  a  serious  event  of  emergency  occur 
during  the  conduct  of  the  research,  I  understand  that  emergency  first  aid  facilities  will  be  on 
hand  to  deal  with  any  physical  problems.  Should  any  problems  of  a  psychological  nature 
arise,  I  acknowledge  that  the  matter  will  be  immediately  forwarded  to  a  relevant  and 
qualified  health  care  individual  for  treatment. 

I  understand  that  any  information  I  provide  is  confidential,  and  that  no  information  that 
could  lead  to  the  identification  of  any  individual  will  be  disclosed  in  any  reports  on  the 
project,  or  to  any  other  party. 

I  understand  that  I  will  be  given  a  transcript  of  data  concerning  me  for  my  approval  before  it 
is  included  in  the  write  up  of  the  research.  I  understand  that  data  collected  will  be  retained 
for  a  period  of  5  years. 

I  also  understand  that  my  participation  is  voluntary,  that  I  can  choose  not  to  participate  in 
part  or  all  of  the  project,  and  that  I  can  withdraw  at  any  stage  of  the  project  without  being 
penalised  or  disadvantaged  in  any  way. 


Signature 


Date 
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Appendix  F:  Task  Description 


F.l.  Task 

The  aim  of  this  research  is  to  investigate  the  effect  of  different  user  interface  designs  on  the 
performance  of  an  Unmanned  Underwater  Vehicle  (UUV)  operator/ pilot.  This  experiment 
will  involve  presenting  participants  with  three  competing  displays  implemented  within  a 
simulated  environment. 

Using  the  simulation  environment,  participants  will  be  asked  to  fly  a  mission  using 
information  provided  to  them  both  verbally  by  the  researcher,  and  visually  by  their  Head- 
Up  operator  display.  Human  performance  and  Situation  Awareness  data  will  be  collected. 
Human  performance  data  will  be  collected  directly  from  the  simulation  program  and 
logged.  Situation  Awareness  data  will  be  collected  electronically  by  a  series  of  probe 
questions  asked  about  each  experimental  condition. 

Participants  will  be  given  a  five  minute  practice  session  to  familiarise  themselves  with 
joystick  controls  prior  to  commencement  of  the  experimental  conditions.  You  will  be 
provided  with  a  description  of  the  joystick  controls  for  use  during  the  practice  session.  It  is 
advised  that  you  familiarise  yourself  with  the  joystick  and  all  of  the  vehicles  movements  in 
order  see  how  the  vehicle  will  respond  to  your  input.  You  will  be  asked  to  use  the  vehicle 
in  all  degrees  of  freedom  during  the  experimental  conditions.  At  the  conclusion  of  the  5 
minute  familiarisation  session,  the  researcher  will  initiate  the  first  of  the  three  scenarios. 
Participants  will  be  given  15  minutes  to  fly  each  scenario.  You  may  fly  at  what  ever  speed 
you  feel  comfortable. 

The  experimental  task  will  require  you  to  fly  towards  a  set  of  waypoints  located  on  a  chart 
in  the  top  right  hand  corner  of  the  operator  screen.  You  must  use  the  instruments  and 
information  provided  to  you  on  the  operator  screen  to  effectively  manoeuvre  the  vehicle 
towards  the  waypoints.  The  researcher  will  provide  you  verbally  with  a  relative  heading 
for  each  waypoint,  however  this  will  be  done  only  once  for  each  waypoint.  You  will  need 
to  follow  the  researchers  heading  initially  in  order  to  progress  towards  each  of  the 
waypoints.  Participants  must  fly  towards  each  waypoint  and  pass  as  close  as  possible  to  it, 
if  possible  flying  through  it.  Participants  must  acknowledge  to  the  researcher  when  they 
establish  visual  contact.  When  the  participant  acknowledges  visual  contact  of  the 
waypoint,  the  researcher  will  provide  the  participant  with  a  relative  heading  to  the  next 
waypoint.  Participants  should  commence  along  that  heading,  making  adjustments  where 
necessary  to  reach  the  next  waypoint.  The  vehicle  is  restricted  to  travel  below  the  water 
surface  so  reaching  waypoints  may  require  the  operator  to  navigate  around  and/  or  over 
objects  in  order  to  reach  each  waypoint. 

Waypoints  will  become  visible  when  the  circular  vehicle  indicator  overlays  the  red  cross  in 
the  chart  at  the  top  right  hand  corner  of  the  operator  screen.  The  circular  vehicle  indicator 
will  overlay  the  cross  if  you  are  in  the  right  area.  You  should  be  able  to  see  the  waypoint 
when  you  are  approaching  it  if  the  vehicle  circle  is  overlaying  the  red  cross  somewhat. 
Waypoints  will  not  necessarily  be  positioned  on  the  ground.  They  may  be  located  in  the 
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water  column  above  you.  If  you  do  not  immediately  see  the  waypoint  as  your  vehicle 
mark  approaches  the  waypoint  cross  on  the  reference  chart,  participants  should  survey  the 
surrounding  area  both  above  and  below  them  to  locate  the  waypoint. 

The  aim  of  the  task  is  to  use  the  instruments  provided  to  help  you  reach  the  waypoints. 
Participants  will  be  required  to  utilise  the  instruments  provided  to  help  them  gain 
Situation  Awareness  and  to  enable  them  to  sufficiently  maneuver  the  vehicle  towards  each 
of  the  waypoints.  In  order  to  assess  a  participants  level  of  Situation  Awareness,  at  random 
intervals  during  each  scenario,  the  simulation  will  be  paused  and  participants  will  be 
asked  a  series  of  Situation  Awareness  questions  relating  to  their  Head-Up  instruments  and 
where  they  are  within  their  environment.  The  operator  display  will  be  temporarily 
blanked  out  and  questions  will  be  presented  to  assess  the  participants  level  of  Situation 
Awareness  (Situation  Awareness  defined  as  "the  perception  of  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the  projection  of  the 
status  of  these  elements  into  the  near  future"  [17,  18].  When  prompted  with  a  Situation 
Awareness  question,  please  verbalise  your  answer  to  the  researcher  so  they  can  enter  your 
answer  in  the  box  on  the  screen.  Once  your  answer  is  logged,  the  scenario  will  continue. 
The  aim  of  the  experiment  is  to  utilise  your  instruments  to  make  sure  you  are  where  you 
are  meant  to  be  to  reach  the  next  waypoint.  Try  to  reach  as  many  waypoints  as  you  can 
during  the  15  minute  time  limit.  Be  aware  that  you  will  not  reach  all  waypoints  in  the  time 
permitted. 

The  vehicle  will  be  initially  neutrally  positioned  in  terms  of  roll.  Try  to  keep  the  vehicle  as 
flat  as  you  can.  Be  aware  that  the  vehicle  is  highly  maneuverable  and  that  it  is  very  easy  to 
flip  (both  pitch  and  roll)  the  vehicle.  Participants  should  also  keep  the  vehicle  as  close  as 
possible  to  between  10-20  metres  to  the  seabed  at  all  times. 


Good  luck! 
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